[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:37:17 dist-test-slave-32hj krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:37:17 dist-test-slave-32hj krb5kdc[399](info): set up 2 sockets
Jun 19 14:37:17 dist-test-slave-32hj krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260619 14:37:18.632640 392 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/krb5kdc/test-user.keytab.
Jun 19 14:37:18 dist-test-slave-32hj krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:37:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:37:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:37:18.743963 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40907
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40907
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:37:18.848397 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:37:18.848639 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:37:18.848678 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:37:18.852608 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:37:18.852659 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:37:18.852676 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:37:18.852691 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:37:18.852707 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:37:18.857834 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40907
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:37:18.858791 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:37:18.859747 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:37:18.867983 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:18.868006 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:18.868006 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:18.868177 412 server_base.cc:1034] running on GCE node
I20260619 14:37:18.868758 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:37:18.869824 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:37:18.870995 412 hybrid_clock.cc:648] HybridClock initialized: now 1781879838870938 us; error 67 us; skew 500 ppm
Jun 19 14:37:18 dist-test-slave-32hj krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:37:18.873662 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:37:18.874498 412 webserver.cc:460] Webserver started at http://127.0.98.62:44765/ using document root <none> and password file <none>
I20260619 14:37:18.875006 412 fs_manager.cc:359] Metadata directory not provided
I20260619 14:37:18.875065 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:37:18.875254 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:37:18.877274 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/master-0/data/instance:
uuid: "ec025090665c4ba28c60f04c5e160f32"
format_stamp: "Formatted at 2026-06-19 14:37:18 on dist-test-slave-32hj"
server_key: "bc2c3e55b30dddd070ffe44a1ef91b7e"
server_key_iv: "2e426a7ca99f21d01389c555e3bfc02e"
server_key_version: "encryptionkey@0"
I20260619 14:37:18.877805 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/master-0/wal/instance:
uuid: "ec025090665c4ba28c60f04c5e160f32"
format_stamp: "Formatted at 2026-06-19 14:37:18 on dist-test-slave-32hj"
server_key: "bc2c3e55b30dddd070ffe44a1ef91b7e"
server_key_iv: "2e426a7ca99f21d01389c555e3bfc02e"
server_key_version: "encryptionkey@0"
I20260619 14:37:18.881372 412 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:37:18.883580 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:18.884734 412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:37:18.884842 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/master-0/wal
uuid: "ec025090665c4ba28c60f04c5e160f32"
format_stamp: "Formatted at 2026-06-19 14:37:18 on dist-test-slave-32hj"
server_key: "bc2c3e55b30dddd070ffe44a1ef91b7e"
server_key_iv: "2e426a7ca99f21d01389c555e3bfc02e"
server_key_version: "encryptionkey@0"
I20260619 14:37:18.884930 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:37:18.898232 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:37:18.900779 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:37:18.900950 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:37:18.906955 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40907
I20260619 14:37:18.906996 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40907 every 8 connection(s)
I20260619 14:37:18.907800 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:37:18.909617 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 412
I20260619 14:37:18.909767 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/master-0/wal/instance
I20260619 14:37:18.910771 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:18.911320 392 external_mini_cluster.cc:1428] Setting key 9606147f9927f7fa5ad5ce6034d33154
I20260619 14:37:18.916865 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec025090665c4ba28c60f04c5e160f32: Bootstrap starting.
I20260619 14:37:18.919023 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec025090665c4ba28c60f04c5e160f32: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:18.919855 480 log.cc:824] T 00000000000000000000000000000000 P ec025090665c4ba28c60f04c5e160f32: Log is configured to *not* fsync() on all Append() calls
I20260619 14:37:18.921643 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec025090665c4ba28c60f04c5e160f32: No bootstrap required, opened a new log
I20260619 14:37:18.925643 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec025090665c4ba28c60f04c5e160f32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec025090665c4ba28c60f04c5e160f32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40907 } }
I20260619 14:37:18.925899 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec025090665c4ba28c60f04c5e160f32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:18.925954 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec025090665c4ba28c60f04c5e160f32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec025090665c4ba28c60f04c5e160f32, State: Initialized, Role: FOLLOWER
I20260619 14:37:18.926347 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec025090665c4ba28c60f04c5e160f32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec025090665c4ba28c60f04c5e160f32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40907 } }
I20260619 14:37:18.926465 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec025090665c4ba28c60f04c5e160f32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:37:18.926520 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec025090665c4ba28c60f04c5e160f32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:37:18.926606 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec025090665c4ba28c60f04c5e160f32 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:18.927490 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec025090665c4ba28c60f04c5e160f32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec025090665c4ba28c60f04c5e160f32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40907 } }
I20260619 14:37:18.927802 480 leader_election.cc:302] T 00000000000000000000000000000000 P ec025090665c4ba28c60f04c5e160f32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec025090665c4ba28c60f04c5e160f32; no voters:
I20260619 14:37:18.928072 480 leader_election.cc:288] T 00000000000000000000000000000000 P ec025090665c4ba28c60f04c5e160f32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:37:18.928143 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec025090665c4ba28c60f04c5e160f32 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:37:18.928313 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec025090665c4ba28c60f04c5e160f32 [term 1 LEADER]: Becoming Leader. State: Replica: ec025090665c4ba28c60f04c5e160f32, State: Running, Role: LEADER
I20260619 14:37:18.928597 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec025090665c4ba28c60f04c5e160f32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec025090665c4ba28c60f04c5e160f32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40907 } }
Jun 19 14:37:18 dist-test-slave-32hj krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:37:18.929286 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec025090665c4ba28c60f04c5e160f32 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:37:18.929747 485 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec025090665c4ba28c60f04c5e160f32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec025090665c4ba28c60f04c5e160f32. Latest consensus state: current_term: 1 leader_uuid: "ec025090665c4ba28c60f04c5e160f32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec025090665c4ba28c60f04c5e160f32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40907 } } }
I20260619 14:37:18.929849 485 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec025090665c4ba28c60f04c5e160f32 [sys.catalog]: This master's current role is: LEADER
I20260619 14:37:18.931653 489 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:37:18.931995 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec025090665c4ba28c60f04c5e160f32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec025090665c4ba28c60f04c5e160f32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec025090665c4ba28c60f04c5e160f32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40907 } } }
I20260619 14:37:18.932348 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec025090665c4ba28c60f04c5e160f32 [sys.catalog]: This master's current role is: LEADER
I20260619 14:37:18.935151 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:18.914704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44524 (local address 127.0.98.62:40907)
0619 14:37:18.915138 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:18.915158 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:18.916120 (+ 962us) server_negotiation.cc:407] Connection header received
0619 14:37:18.919319 (+ 3199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:18.919338 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:18.919714 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:18.920050 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:18.922161 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:18.923622 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:18.924838 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:18.925525 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:18.925679 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:18.929597 (+ 3918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:18.929604 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:18.929619 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:18.929650 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:18.931720 (+ 2070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:18.932193 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:18.932199 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:18.932204 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:18.932346 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:18.932694 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:18.932698 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:18.932700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:18.933226 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:18.933420 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:18.934013 (+ 593us) server_negotiation.cc:299] Negotiation successful
0619 14:37:18.934281 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":131,"threads_started":1}
I20260619 14:37:18.935177 489 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:37:18.939838 489 catalog_manager.cc:1347] Generated new cluster ID: 46d77a32081c47ae846ef7623a03cc3f
I20260619 14:37:18.939903 489 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:37:18.957367 489 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:37:18.958604 489 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:37:18.972297 489 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec025090665c4ba28c60f04c5e160f32: Generated new TSK 0
I20260619 14:37:18.973155 489 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 14:37:19.007815 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40907
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:37:19.116158 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:37:19.116438 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:37:19.116500 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:37:19.120549 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:37:19.120620 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:37:19.120700 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:37:19.126078 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40907
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:37:19.127135 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:37:19.128122 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:37:19.137622 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:19.137630 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:19.137799 506 server_base.cc:1034] running on GCE node
W20260619 14:37:19.137622 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:19.138401 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:37:19.139024 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:37:19.140201 506 hybrid_clock.cc:648] HybridClock initialized: now 1781879839140183 us; error 34 us; skew 500 ppm
Jun 19 14:37:19 dist-test-slave-32hj krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879839, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:37:19.142895 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:37:19.143748 506 webserver.cc:460] Webserver started at http://127.0.98.1:41521/ using document root <none> and password file <none>
I20260619 14:37:19.144281 506 fs_manager.cc:359] Metadata directory not provided
I20260619 14:37:19.144348 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:37:19.144531 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:37:19.146584 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/ts-0/data/instance:
uuid: "7ac60fe21a004dc4b80a957a75d0657a"
format_stamp: "Formatted at 2026-06-19 14:37:19 on dist-test-slave-32hj"
server_key: "7417bf0ba69dbd5280000f5ffc0e2a84"
server_key_iv: "bba76a9d0ee9f07be51c73823e5586dc"
server_key_version: "encryptionkey@0"
I20260619 14:37:19.147116 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7ac60fe21a004dc4b80a957a75d0657a"
format_stamp: "Formatted at 2026-06-19 14:37:19 on dist-test-slave-32hj"
server_key: "7417bf0ba69dbd5280000f5ffc0e2a84"
server_key_iv: "bba76a9d0ee9f07be51c73823e5586dc"
server_key_version: "encryptionkey@0"
I20260619 14:37:19.150791 506 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:37:19.153407 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:19.154711 506 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:37:19.154809 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/ts-0/wal
uuid: "7ac60fe21a004dc4b80a957a75d0657a"
format_stamp: "Formatted at 2026-06-19 14:37:19 on dist-test-slave-32hj"
server_key: "7417bf0ba69dbd5280000f5ffc0e2a84"
server_key_iv: "bba76a9d0ee9f07be51c73823e5586dc"
server_key_version: "encryptionkey@0"
I20260619 14:37:19.154917 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:37:19.176223 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:37:19.179009 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:37:19.179174 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:37:19.179742 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:37:19.180565 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:37:19.180612 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:19.180652 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:37:19.180681 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:19.189074 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38413
I20260619 14:37:19.189105 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38413 every 8 connection(s)
I20260619 14:37:19.189935 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:37:19.193522 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 506
I20260619 14:37:19.193629 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:37:19.193858 392 external_mini_cluster.cc:1428] Setting key 5e3d95218cb79778aa2a2575d62400ae
Jun 19 14:37:19 dist-test-slave-32hj krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879839, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:37:19.203249 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:19.191622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48341 (local address 127.0.98.62:40907)
0619 14:37:19.191758 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:19.191763 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:19.192605 (+ 842us) server_negotiation.cc:407] Connection header received
0619 14:37:19.193953 (+ 1348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:19.193957 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:19.194011 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:19.194104 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:19.195368 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:19.196105 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:19.197228 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:19.197462 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:19.197549 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:19.199910 (+ 2361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:19.199914 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:19.199917 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:19.199945 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:19.201430 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:19.201860 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:19.201863 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:19.201865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:19.201956 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:19.202224 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:19.202227 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:19.202229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:19.202443 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:19.202539 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:19.202959 (+ 420us) server_negotiation.cc:299] Negotiation successful
0619 14:37:19.203086 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 14:37:19.203922 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:19.192031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40907 (local address 127.0.98.1:48341)
0619 14:37:19.192462 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:19.192503 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:19.193752 (+ 1249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:19.194249 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:19.194256 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:19.194643 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:19.195232 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:19.195247 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:19.196202 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:19.196204 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:19.197119 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:19.197127 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:19.197592 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:19.197596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:19.197746 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:19.198247 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:19.198262 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:19.199766 (+ 1504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:19.201530 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:19.201534 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:19.201545 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:19.201772 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:19.202035 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:19.202037 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:19.202039 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:19.202149 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:19.202525 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:19.202529 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:19.202738 (+ 209us) client_negotiation.cc:769] Sending connection context
0619 14:37:19.202882 (+ 144us) client_negotiation.cc:240] Negotiation successful
0619 14:37:19.203107 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":108,"threads_started":1}
I20260619 14:37:19.205152 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40907
I20260619 14:37:19.205367 635 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 14:37:19.205842 635 heartbeater.cc:511] Master 127.0.98.62:40907 requested a full tablet report, sending...
I20260619 14:37:19.207261 444 ts_manager.cc:194] Registered new tserver with Master: 7ac60fe21a004dc4b80a957a75d0657a (127.0.98.1:38413)
I20260619 14:37:19.208312 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48341
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 14:37:19.218909 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40907
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:37:19.324776 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:37:19.325030 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:37:19.325073 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:37:19.329005 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:37:19.329063 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:37:19.329123 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:37:19.334564 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40907
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:37:19.335472 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:37:19.336371 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:37:19.345148 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:19.345160 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:19.345160 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:19.345304 640 server_base.cc:1034] running on GCE node
I20260619 14:37:19.345772 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:37:19.346315 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:37:19.347469 640 hybrid_clock.cc:648] HybridClock initialized: now 1781879839347452 us; error 26 us; skew 500 ppm
Jun 19 14:37:19 dist-test-slave-32hj krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879839, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:37:19.349915 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:37:19.350693 640 webserver.cc:460] Webserver started at http://127.0.98.2:37287/ using document root <none> and password file <none>
I20260619 14:37:19.351199 640 fs_manager.cc:359] Metadata directory not provided
I20260619 14:37:19.351262 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:37:19.351435 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:37:19.353462 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/ts-1/data/instance:
uuid: "6c9db1b9be1049d5a736c41868e3e238"
format_stamp: "Formatted at 2026-06-19 14:37:19 on dist-test-slave-32hj"
server_key: "eb8b33a42f9520ddce2f3fb6a253736b"
server_key_iv: "0b07d5323e70050b80da0ba2b5a56dc1"
server_key_version: "encryptionkey@0"
I20260619 14:37:19.353979 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6c9db1b9be1049d5a736c41868e3e238"
format_stamp: "Formatted at 2026-06-19 14:37:19 on dist-test-slave-32hj"
server_key: "eb8b33a42f9520ddce2f3fb6a253736b"
server_key_iv: "0b07d5323e70050b80da0ba2b5a56dc1"
server_key_version: "encryptionkey@0"
I20260619 14:37:19.357466 640 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:37:19.359803 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:19.361021 640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:37:19.361119 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/ts-1/wal
uuid: "6c9db1b9be1049d5a736c41868e3e238"
format_stamp: "Formatted at 2026-06-19 14:37:19 on dist-test-slave-32hj"
server_key: "eb8b33a42f9520ddce2f3fb6a253736b"
server_key_iv: "0b07d5323e70050b80da0ba2b5a56dc1"
server_key_version: "encryptionkey@0"
I20260619 14:37:19.361203 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:37:19.381729 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:37:19.384644 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:37:19.384825 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:37:19.385376 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:37:19.386127 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:37:19.386171 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:19.386221 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:37:19.386246 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:19.395051 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33253
I20260619 14:37:19.395079 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33253 every 8 connection(s)
I20260619 14:37:19.395841 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:37:19.404276 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 640
I20260619 14:37:19.404376 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/ts-1/wal/instance
Jun 19 14:37:19 dist-test-slave-32hj krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879839, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:37:19.404629 392 external_mini_cluster.cc:1428] Setting key c1a1198e05bf0af7e405159c88795941
I20260619 14:37:19.408375 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:19.397507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51925 (local address 127.0.98.62:40907)
0619 14:37:19.397621 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:19.397625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:19.398396 (+ 771us) server_negotiation.cc:407] Connection header received
0619 14:37:19.399649 (+ 1253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:19.399652 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:19.399709 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:19.399778 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:19.400910 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:19.401434 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:19.402520 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:19.402735 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:19.402819 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:19.405205 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:19.405209 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:19.405212 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:19.405237 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:19.406494 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:19.407018 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:19.407020 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:19.407021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:19.407087 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:19.407400 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:19.407403 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:19.407405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:19.407564 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:19.407649 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:19.408112 (+ 463us) server_negotiation.cc:299] Negotiation successful
0619 14:37:19.408225 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 14:37:19.409202 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:19.397845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40907 (local address 127.0.98.2:51925)
0619 14:37:19.398253 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:19.398294 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:19.399446 (+ 1152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:19.399958 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:19.399965 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:19.400284 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:19.400771 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:19.400786 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:19.401547 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:19.401550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:19.402401 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:19.402410 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:19.402863 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:19.402867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:19.403024 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:19.403537 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:19.403553 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:19.405044 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:19.406602 (+ 1558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:19.406607 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:19.406621 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:19.406936 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:19.407154 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:19.407156 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:19.407159 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:19.407319 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:19.407665 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:19.407671 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:19.407895 (+ 224us) client_negotiation.cc:769] Sending connection context
0619 14:37:19.408057 (+ 162us) client_negotiation.cc:240] Negotiation successful
0619 14:37:19.408354 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":106,"threads_started":1}
I20260619 14:37:19.410633 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40907
I20260619 14:37:19.410880 769 heartbeater.cc:464] Registering TS with master...
I20260619 14:37:19.411451 769 heartbeater.cc:511] Master 127.0.98.62:40907 requested a full tablet report, sending...
I20260619 14:37:19.412467 444 ts_manager.cc:194] Registered new tserver with Master: 6c9db1b9be1049d5a736c41868e3e238 (127.0.98.2:33253)
I20260619 14:37:19.413055 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51925
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 14:37:19.428215 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40907
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:37:19.536002 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:37:19.536253 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:37:19.536296 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:37:19.540216 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:37:19.540274 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:37:19.540333 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:37:19.545547 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40907
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:37:19.546499 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:37:19.547418 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:37:19.556828 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:19.556839 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:19.556836 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:19.556885 774 server_base.cc:1034] running on GCE node
I20260619 14:37:19.557447 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:37:19.558046 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:37:19.559196 774 hybrid_clock.cc:648] HybridClock initialized: now 1781879839559169 us; error 41 us; skew 500 ppm
Jun 19 14:37:19 dist-test-slave-32hj krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879839, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:37:19.561754 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:37:19.562532 774 webserver.cc:460] Webserver started at http://127.0.98.3:38819/ using document root <none> and password file <none>
I20260619 14:37:19.563061 774 fs_manager.cc:359] Metadata directory not provided
I20260619 14:37:19.563123 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:37:19.563298 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:37:19.565272 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/ts-2/data/instance:
uuid: "2d105aee33cb428c86946b2203e5c2c3"
format_stamp: "Formatted at 2026-06-19 14:37:19 on dist-test-slave-32hj"
server_key: "d2fb9cfb961e90c6dd3eb319947a6b65"
server_key_iv: "5c64d0d6ac99a706b4ea2960c75df4b4"
server_key_version: "encryptionkey@0"
I20260619 14:37:19.565789 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2d105aee33cb428c86946b2203e5c2c3"
format_stamp: "Formatted at 2026-06-19 14:37:19 on dist-test-slave-32hj"
server_key: "d2fb9cfb961e90c6dd3eb319947a6b65"
server_key_iv: "5c64d0d6ac99a706b4ea2960c75df4b4"
server_key_version: "encryptionkey@0"
I20260619 14:37:19.569191 774 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:37:19.571434 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:19.572610 774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:37:19.572710 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/ts-2/wal
uuid: "2d105aee33cb428c86946b2203e5c2c3"
format_stamp: "Formatted at 2026-06-19 14:37:19 on dist-test-slave-32hj"
server_key: "d2fb9cfb961e90c6dd3eb319947a6b65"
server_key_iv: "5c64d0d6ac99a706b4ea2960c75df4b4"
server_key_version: "encryptionkey@0"
I20260619 14:37:19.572844 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:37:19.600399 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:37:19.603493 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:37:19.603657 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:37:19.604195 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:37:19.604992 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:37:19.605046 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:19.605098 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:37:19.605123 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:19.613595 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43569
I20260619 14:37:19.613619 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43569 every 8 connection(s)
I20260619 14:37:19.614423 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
Jun 19 14:37:19 dist-test-slave-32hj krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879839, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:37:19.623920 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 774
I20260619 14:37:19.624053 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781879837648036-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:37:19.624311 392 external_mini_cluster.cc:1428] Setting key f8d1b6d1bc34baecf7149933be50414f
I20260619 14:37:19.626901 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:19.616021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33603 (local address 127.0.98.62:40907)
0619 14:37:19.616139 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:19.616142 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:19.616949 (+ 807us) server_negotiation.cc:407] Connection header received
0619 14:37:19.618202 (+ 1253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:19.618205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:19.618261 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:19.618339 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:19.619452 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:19.619973 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:19.621096 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:19.621310 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:19.621396 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:19.623559 (+ 2163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:19.623562 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:19.623565 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:19.623589 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:19.625083 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:19.625538 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:19.625541 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:19.625543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:19.625619 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:19.625911 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:19.625914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:19.625916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:19.626086 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:19.626170 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:19.626629 (+ 459us) server_negotiation.cc:299] Negotiation successful
0619 14:37:19.626749 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 14:37:19.627631 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:19.616379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40907 (local address 127.0.98.3:33603)
0619 14:37:19.616811 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:19.616852 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:19.617999 (+ 1147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:19.618529 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:19.618536 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:19.618866 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:19.619310 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:19.619322 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:19.620084 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:19.620087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:19.620976 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:19.620985 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:19.621446 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:19.621450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:19.621598 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:19.622078 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:19.622093 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:19.623409 (+ 1316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:19.625179 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:19.625185 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:19.625208 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:19.625447 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:19.625695 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:19.625698 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:19.625700 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:19.625828 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:19.626177 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:19.626183 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:19.626419 (+ 236us) client_negotiation.cc:769] Sending connection context
0619 14:37:19.626574 (+ 155us) client_negotiation.cc:240] Negotiation successful
0619 14:37:19.626871 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":102,"threads_started":1}
I20260619 14:37:19.628669 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40907
I20260619 14:37:19.628901 903 heartbeater.cc:464] Registering TS with master...
I20260619 14:37:19.629356 903 heartbeater.cc:511] Master 127.0.98.62:40907 requested a full tablet report, sending...
I20260619 14:37:19.630293 444 ts_manager.cc:194] Registered new tserver with Master: 2d105aee33cb428c86946b2203e5c2c3 (127.0.98.3:43569)
I20260619 14:37:19.630843 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33603
I20260619 14:37:19.638268 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 14:37:19 dist-test-slave-32hj krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879839, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 14:37:19 dist-test-slave-32hj krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879839, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 14:37:19.662856 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:19.654003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48064 (local address 127.0.98.1:38413)
0619 14:37:19.654260 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:19.654275 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:19.654301 (+ 26us) server_negotiation.cc:407] Connection header received
0619 14:37:19.654416 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:19.654423 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:19.654615 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:19.654742 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:19.655379 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:19.656131 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:19.656981 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:19.657342 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:19.657434 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:19.659692 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:19.659700 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:19.659712 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:19.659735 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:19.661224 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:19.661606 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:19.661612 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:19.661613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:19.661691 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:19.661929 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:19.661932 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:19.661934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:19.662160 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:19.662353 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:19.662555 (+ 202us) server_negotiation.cc:299] Negotiation successful
0619 14:37:19.662699 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":63,"threads_started":1}
W20260619 14:37:19.663794 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48064
Jun 19 14:37:19 dist-test-slave-32hj krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 14:37:19 dist-test-slave-32hj krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 14:37:19.687883 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:19.679982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48072 (local address 127.0.98.1:38413)
0619 14:37:19.680137 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:19.680141 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:19.680212 (+ 71us) server_negotiation.cc:407] Connection header received
0619 14:37:19.680324 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:19.680327 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:19.680376 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:19.680453 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:19.681234 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:19.681745 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:19.682574 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:19.682785 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:19.682866 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:19.685052 (+ 2186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:19.685055 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:19.685057 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:19.685081 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:19.686435 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:19.686817 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:19.686820 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:19.686821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:19.686891 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:19.687123 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:19.687126 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:19.687128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:19.687281 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:19.687390 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:19.687615 (+ 225us) server_negotiation.cc:299] Negotiation successful
0619 14:37:19.687722 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 14:37:19.688896 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 506
I20260619 14:37:19.693964 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 640
I20260619 14:37:19.700250 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 774
I20260619 14:37:19.704604 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 412
2026-06-19T14:37:19Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2048 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:37:19 dist-test-slave-32hj krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:37:19 dist-test-slave-32hj krb5kdc[917](info): set up 2 sockets
Jun 19 14:37:19 dist-test-slave-32hj krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260619 14:37:20.660100 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/krb5kdc/test-user.keytab.
Jun 19 14:37:20 dist-test-slave-32hj krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:37:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:37:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:37:20.765799 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46301
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46301
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:37:20.869767 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:37:20.870033 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:37:20.870071 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:37:20.873977 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:37:20.874029 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:37:20.874047 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:37:20.874061 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:37:20.874074 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:37:20.879082 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46301
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:37:20.879976 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:37:20.880901 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:37:20.888665 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:20.888652 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:20.888676 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:20.889006 930 server_base.cc:1034] running on GCE node
I20260619 14:37:20.889504 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:37:20.890547 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:37:20.891703 930 hybrid_clock.cc:648] HybridClock initialized: now 1781879840891687 us; error 31 us; skew 500 ppm
Jun 19 14:37:20 dist-test-slave-32hj krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879840, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:37:20.898564 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:37:20.899397 930 webserver.cc:460] Webserver started at http://127.0.98.62:46497/ using document root <none> and password file <none>
I20260619 14:37:20.899899 930 fs_manager.cc:359] Metadata directory not provided
I20260619 14:37:20.899961 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:37:20.900138 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:37:20.902104 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/master-0/data/instance:
uuid: "8e1bc1ce07854556902ce1b1d837236e"
format_stamp: "Formatted at 2026-06-19 14:37:20 on dist-test-slave-32hj"
server_key: "d7a93f0736ec680d27a550d1c9f30140"
server_key_iv: "947bff5b4cc95462ea38acbb65b6ee9c"
server_key_version: "encryptionkey@0"
I20260619 14:37:20.902601 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/master-0/wal/instance:
uuid: "8e1bc1ce07854556902ce1b1d837236e"
format_stamp: "Formatted at 2026-06-19 14:37:20 on dist-test-slave-32hj"
server_key: "d7a93f0736ec680d27a550d1c9f30140"
server_key_iv: "947bff5b4cc95462ea38acbb65b6ee9c"
server_key_version: "encryptionkey@0"
I20260619 14:37:20.906004 930 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:37:20.908241 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:20.909396 930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:37:20.909508 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/master-0/wal
uuid: "8e1bc1ce07854556902ce1b1d837236e"
format_stamp: "Formatted at 2026-06-19 14:37:20 on dist-test-slave-32hj"
server_key: "d7a93f0736ec680d27a550d1c9f30140"
server_key_iv: "947bff5b4cc95462ea38acbb65b6ee9c"
server_key_version: "encryptionkey@0"
I20260619 14:37:20.909596 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:37:20.926326 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:37:20.928963 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:37:20.929124 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:37:20.935382 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46301
I20260619 14:37:20.935402 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46301 every 8 connection(s)
I20260619 14:37:20.936240 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:37:20.938782 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:20.941051 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 930
I20260619 14:37:20.941160 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/master-0/wal/instance
I20260619 14:37:20.941416 392 external_mini_cluster.cc:1428] Setting key fd83152d1cc642270d8f7afbe3d92b6a
I20260619 14:37:20.944509 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e1bc1ce07854556902ce1b1d837236e: Bootstrap starting.
I20260619 14:37:20.946619 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e1bc1ce07854556902ce1b1d837236e: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:20.947316 998 log.cc:824] T 00000000000000000000000000000000 P 8e1bc1ce07854556902ce1b1d837236e: Log is configured to *not* fsync() on all Append() calls
I20260619 14:37:20.949141 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e1bc1ce07854556902ce1b1d837236e: No bootstrap required, opened a new log
Jun 19 14:37:20 dist-test-slave-32hj krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:37:20.952103 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e1bc1ce07854556902ce1b1d837236e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e1bc1ce07854556902ce1b1d837236e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46301 } }
I20260619 14:37:20.952325 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e1bc1ce07854556902ce1b1d837236e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:20.952360 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e1bc1ce07854556902ce1b1d837236e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e1bc1ce07854556902ce1b1d837236e, State: Initialized, Role: FOLLOWER
I20260619 14:37:20.952823 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e1bc1ce07854556902ce1b1d837236e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e1bc1ce07854556902ce1b1d837236e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46301 } }
I20260619 14:37:20.952975 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e1bc1ce07854556902ce1b1d837236e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:37:20.953039 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e1bc1ce07854556902ce1b1d837236e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:37:20.953135 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e1bc1ce07854556902ce1b1d837236e [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:20.953954 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e1bc1ce07854556902ce1b1d837236e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e1bc1ce07854556902ce1b1d837236e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46301 } }
I20260619 14:37:20.954274 998 leader_election.cc:302] T 00000000000000000000000000000000 P 8e1bc1ce07854556902ce1b1d837236e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e1bc1ce07854556902ce1b1d837236e; no voters:
I20260619 14:37:20.954537 998 leader_election.cc:288] T 00000000000000000000000000000000 P 8e1bc1ce07854556902ce1b1d837236e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:37:20.954629 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e1bc1ce07854556902ce1b1d837236e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:37:20.954847 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e1bc1ce07854556902ce1b1d837236e [term 1 LEADER]: Becoming Leader. State: Replica: 8e1bc1ce07854556902ce1b1d837236e, State: Running, Role: LEADER
I20260619 14:37:20.955200 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e1bc1ce07854556902ce1b1d837236e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e1bc1ce07854556902ce1b1d837236e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46301 } }
I20260619 14:37:20.955273 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:20.942650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39588 (local address 127.0.98.62:46301)
0619 14:37:20.943186 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:20.943201 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:20.943236 (+ 35us) server_negotiation.cc:407] Connection header received
0619 14:37:20.943852 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:20.943868 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:20.944183 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:20.944500 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:20.945279 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:20.946254 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:20.947117 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:20.947617 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:20.947760 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:20.950270 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:20.950275 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:20.950288 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:20.950314 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:20.952325 (+ 2011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:20.952737 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:20.952743 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:20.952749 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:20.952862 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:20.953113 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:20.953117 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:20.953119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:20.953542 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:20.953725 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:20.953986 (+ 261us) server_negotiation.cc:299] Negotiation successful
0619 14:37:20.954281 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":231,"threads_started":1}
I20260619 14:37:20.955721 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e1bc1ce07854556902ce1b1d837236e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:37:20.956416 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e1bc1ce07854556902ce1b1d837236e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e1bc1ce07854556902ce1b1d837236e. Latest consensus state: current_term: 1 leader_uuid: "8e1bc1ce07854556902ce1b1d837236e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e1bc1ce07854556902ce1b1d837236e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46301 } } }
I20260619 14:37:20.956526 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e1bc1ce07854556902ce1b1d837236e [sys.catalog]: This master's current role is: LEADER
I20260619 14:37:20.957376 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e1bc1ce07854556902ce1b1d837236e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e1bc1ce07854556902ce1b1d837236e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e1bc1ce07854556902ce1b1d837236e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46301 } } }
I20260619 14:37:20.957465 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e1bc1ce07854556902ce1b1d837236e [sys.catalog]: This master's current role is: LEADER
I20260619 14:37:20.957752 1011 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:37:20.960763 1011 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:37:20.965487 1011 catalog_manager.cc:1347] Generated new cluster ID: a66b91b1e95e4cf4a2d83fe858a188cb
I20260619 14:37:20.965550 1011 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:37:20.972108 1011 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:37:20.973080 1011 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:37:20.981482 1011 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e1bc1ce07854556902ce1b1d837236e: Generated new TSK 0
I20260619 14:37:20.982060 1011 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 14:37:21.015833 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46301
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:37:21.131626 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:37:21.131880 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:37:21.131923 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:37:21.135900 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:37:21.135962 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:37:21.136020 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:37:21.141196 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46301
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:37:21.142146 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:37:21.143102 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:37:21.151923 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:21.151922 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:21.152089 1024 server_base.cc:1034] running on GCE node
W20260619 14:37:21.151929 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:21.152573 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:37:21.153199 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:37:21.154381 1024 hybrid_clock.cc:648] HybridClock initialized: now 1781879841154357 us; error 32 us; skew 500 ppm
Jun 19 14:37:21 dist-test-slave-32hj krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879841, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:37:21.157233 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:37:21.158149 1024 webserver.cc:460] Webserver started at http://127.0.98.1:41261/ using document root <none> and password file <none>
I20260619 14:37:21.158706 1024 fs_manager.cc:359] Metadata directory not provided
I20260619 14:37:21.158794 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:37:21.158972 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:37:21.160952 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/ts-0/data/instance:
uuid: "ee0b1ab1b4de4808b72c4d5949f92f4a"
format_stamp: "Formatted at 2026-06-19 14:37:21 on dist-test-slave-32hj"
server_key: "98213e67f7597ce4ce838598b52b7e2d"
server_key_iv: "13a8a906d74fda934dbc7cbe5c584932"
server_key_version: "encryptionkey@0"
I20260619 14:37:21.161464 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ee0b1ab1b4de4808b72c4d5949f92f4a"
format_stamp: "Formatted at 2026-06-19 14:37:21 on dist-test-slave-32hj"
server_key: "98213e67f7597ce4ce838598b52b7e2d"
server_key_iv: "13a8a906d74fda934dbc7cbe5c584932"
server_key_version: "encryptionkey@0"
I20260619 14:37:21.164981 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:37:21.167186 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:21.168319 1024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:37:21.168419 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/ts-0/wal
uuid: "ee0b1ab1b4de4808b72c4d5949f92f4a"
format_stamp: "Formatted at 2026-06-19 14:37:21 on dist-test-slave-32hj"
server_key: "98213e67f7597ce4ce838598b52b7e2d"
server_key_iv: "13a8a906d74fda934dbc7cbe5c584932"
server_key_version: "encryptionkey@0"
I20260619 14:37:21.168510 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:37:21.180996 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:37:21.183871 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:37:21.184029 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:37:21.184566 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:37:21.185343 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:37:21.185395 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:21.185433 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:37:21.185449 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:21.193526 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41373
I20260619 14:37:21.193605 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41373 every 8 connection(s)
I20260619 14:37:21.194348 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:37:21.202265 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 1024
I20260619 14:37:21.202364 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:37:21.202651 392 external_mini_cluster.cc:1428] Setting key b20b144ddd7356cee4a9afb29f015407
Jun 19 14:37:21 dist-test-slave-32hj krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879841, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:37:21.207419 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:21.195983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42191 (local address 127.0.98.62:46301)
0619 14:37:21.196095 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:21.196099 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:21.196884 (+ 785us) server_negotiation.cc:407] Connection header received
0619 14:37:21.198139 (+ 1255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:21.198142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:21.198188 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:21.198258 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:21.199380 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:21.199925 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:21.201020 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:21.201274 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:21.201358 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:21.203876 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:21.203880 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:21.203883 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:21.203907 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:21.205390 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:21.205868 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:21.205878 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:21.205880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:21.205972 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:21.206312 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:21.206314 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:21.206316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:21.206530 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:21.206623 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:21.207144 (+ 521us) server_negotiation.cc:299] Negotiation successful
0619 14:37:21.207262 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 14:37:21.208114 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:21.196309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46301 (local address 127.0.98.1:42191)
0619 14:37:21.196704 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:21.196784 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:21.197957 (+ 1173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:21.198402 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:21.198408 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:21.198737 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:21.199247 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:21.199260 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:21.200015 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:21.200017 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:21.200929 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:21.200938 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:21.201390 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:21.201393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:21.201528 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:21.201991 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:21.202006 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:21.203730 (+ 1724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:21.205502 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:21.205507 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:21.205518 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:21.205773 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:21.206063 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:21.206066 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:21.206068 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:21.206228 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:21.206622 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:21.206628 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:21.206920 (+ 292us) client_negotiation.cc:769] Sending connection context
0619 14:37:21.207070 (+ 150us) client_negotiation.cc:240] Negotiation successful
0619 14:37:21.207315 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":98,"threads_started":1}
I20260619 14:37:21.209304 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46301
I20260619 14:37:21.209517 1153 heartbeater.cc:464] Registering TS with master...
I20260619 14:37:21.209959 1153 heartbeater.cc:511] Master 127.0.98.62:46301 requested a full tablet report, sending...
I20260619 14:37:21.211323 962 ts_manager.cc:194] Registered new tserver with Master: ee0b1ab1b4de4808b72c4d5949f92f4a (127.0.98.1:41373)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 14:37:21.212473 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42191
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab.
W20260619 14:37:21.460532 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.2: real 0.256s user 0.000s sys 0.003s
I20260619 14:37:21.461028 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46301
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:37:21.570855 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:37:21.571135 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:37:21.571188 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:37:21.575376 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:37:21.575446 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:37:21.575507 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:37:21.581216 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46301
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:37:21.582264 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:37:21.583261 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:37:21.592760 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:21.592821 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:21.593008 1158 server_base.cc:1034] running on GCE node
W20260619 14:37:21.592800 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:21.593494 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:37:21.594125 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:37:21.595279 1158 hybrid_clock.cc:648] HybridClock initialized: now 1781879841595256 us; error 36 us; skew 500 ppm
Jun 19 14:37:21 dist-test-slave-32hj krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879841, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:37:21.597980 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:37:21.598856 1158 webserver.cc:460] Webserver started at http://127.0.98.2:37085/ using document root <none> and password file <none>
I20260619 14:37:21.599385 1158 fs_manager.cc:359] Metadata directory not provided
I20260619 14:37:21.599449 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:37:21.599633 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:37:21.601644 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/ts-1/data/instance:
uuid: "6e9ba308299f41d6bf99baf56f8df661"
format_stamp: "Formatted at 2026-06-19 14:37:21 on dist-test-slave-32hj"
server_key: "a6292217936c20cf64c3e23f7f0e7bb2"
server_key_iv: "b5d2152cf506627f8959511d3def7491"
server_key_version: "encryptionkey@0"
I20260619 14:37:21.602169 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6e9ba308299f41d6bf99baf56f8df661"
format_stamp: "Formatted at 2026-06-19 14:37:21 on dist-test-slave-32hj"
server_key: "a6292217936c20cf64c3e23f7f0e7bb2"
server_key_iv: "b5d2152cf506627f8959511d3def7491"
server_key_version: "encryptionkey@0"
I20260619 14:37:21.605716 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 14:37:21.608094 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:21.609380 1158 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:37:21.609488 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/ts-1/wal
uuid: "6e9ba308299f41d6bf99baf56f8df661"
format_stamp: "Formatted at 2026-06-19 14:37:21 on dist-test-slave-32hj"
server_key: "a6292217936c20cf64c3e23f7f0e7bb2"
server_key_iv: "b5d2152cf506627f8959511d3def7491"
server_key_version: "encryptionkey@0"
I20260619 14:37:21.609588 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:37:21.624809 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:37:21.627908 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:37:21.628077 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:37:21.628638 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:37:21.629463 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:37:21.629509 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:21.629560 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:37:21.629590 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:21.638072 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43969
I20260619 14:37:21.638109 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43969 every 8 connection(s)
I20260619 14:37:21.638959 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:37:21.647101 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 1158
I20260619 14:37:21.647202 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:37:21.647462 392 external_mini_cluster.cc:1428] Setting key 8c03083db9460ae54ee9c81555245198
Jun 19 14:37:21 dist-test-slave-32hj krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879841, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:37:21.652545 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:21.640653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59269 (local address 127.0.98.62:46301)
0619 14:37:21.640794 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:21.640797 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:21.641666 (+ 869us) server_negotiation.cc:407] Connection header received
0619 14:37:21.643008 (+ 1342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:21.643012 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:21.643069 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:21.643143 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:21.644285 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:21.644869 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:21.645979 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:21.646208 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:21.646295 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:21.648851 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:21.648855 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:21.648858 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:21.648882 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:21.650240 (+ 1358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:21.650903 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:21.650908 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:21.650909 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:21.651000 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:21.651381 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:21.651384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:21.651386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:21.651586 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:21.651675 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:21.652266 (+ 591us) server_negotiation.cc:299] Negotiation successful
0619 14:37:21.652388 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 14:37:21.653973 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:21.641060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46301 (local address 127.0.98.2:59269)
0619 14:37:21.641506 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:21.641550 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:21.642797 (+ 1247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:21.643328 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:21.643335 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:21.643682 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:21.644142 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:21.644155 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:21.644988 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:21.644990 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:21.645855 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:21.645872 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:21.646345 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:21.646349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:21.646520 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:21.647106 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:21.647122 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:21.648669 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:21.650395 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:21.650402 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:21.650419 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:21.650780 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:21.651091 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:21.651094 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:21.651096 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:21.651283 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:21.651697 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:21.651704 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:21.652010 (+ 306us) client_negotiation.cc:769] Sending connection context
0619 14:37:21.652223 (+ 213us) client_negotiation.cc:240] Negotiation successful
0619 14:37:21.652570 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":110,"threads_started":1}
I20260619 14:37:21.655135 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46301
I20260619 14:37:21.655391 1287 heartbeater.cc:464] Registering TS with master...
I20260619 14:37:21.655997 1287 heartbeater.cc:511] Master 127.0.98.62:46301 requested a full tablet report, sending...
I20260619 14:37:21.657196 962 ts_manager.cc:194] Registered new tserver with Master: 6e9ba308299f41d6bf99baf56f8df661 (127.0.98.2:43969)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 14:37:21.657830 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59269
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 14:37:22.214977 1153 heartbeater.cc:503] Master 127.0.98.62:46301 was elected leader, sending a full tablet report...
I20260619 14:37:22.659861 1287 heartbeater.cc:503] Master 127.0.98.62:46301 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab.
W20260619 14:37:22.798523 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.3: real 1.149s user 0.002s sys 0.001s
I20260619 14:37:22.798987 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46301
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:37:22.907109 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:37:22.907377 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:37:22.907419 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:37:22.911288 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:37:22.911338 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:37:22.911398 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:37:22.917022 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46301
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:37:22.917999 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:37:22.918962 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:37:22.927760 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:22.927768 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:22.927924 1292 server_base.cc:1034] running on GCE node
W20260619 14:37:22.927821 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:22.928401 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:37:22.929021 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:37:22.930189 1292 hybrid_clock.cc:648] HybridClock initialized: now 1781879842930175 us; error 24 us; skew 500 ppm
Jun 19 14:37:22 dist-test-slave-32hj krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879842, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:37:22.935992 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:37:22.937047 1292 webserver.cc:460] Webserver started at http://127.0.98.3:34211/ using document root <none> and password file <none>
I20260619 14:37:22.937726 1292 fs_manager.cc:359] Metadata directory not provided
I20260619 14:37:22.937803 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:37:22.938009 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:37:22.940524 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/ts-2/data/instance:
uuid: "a5528a1f1a39481cb005a49cceeecf39"
format_stamp: "Formatted at 2026-06-19 14:37:22 on dist-test-slave-32hj"
server_key: "5d93391bf6737e1b83ca801ae447806e"
server_key_iv: "9d0ae21ea3a1b86caa5fe13878c367d9"
server_key_version: "encryptionkey@0"
I20260619 14:37:22.941087 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a5528a1f1a39481cb005a49cceeecf39"
format_stamp: "Formatted at 2026-06-19 14:37:22 on dist-test-slave-32hj"
server_key: "5d93391bf6737e1b83ca801ae447806e"
server_key_iv: "9d0ae21ea3a1b86caa5fe13878c367d9"
server_key_version: "encryptionkey@0"
I20260619 14:37:22.944873 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:37:22.947223 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:22.948371 1292 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:37:22.948465 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/ts-2/wal
uuid: "a5528a1f1a39481cb005a49cceeecf39"
format_stamp: "Formatted at 2026-06-19 14:37:22 on dist-test-slave-32hj"
server_key: "5d93391bf6737e1b83ca801ae447806e"
server_key_iv: "9d0ae21ea3a1b86caa5fe13878c367d9"
server_key_version: "encryptionkey@0"
I20260619 14:37:22.948565 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:37:22.973623 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:37:22.976683 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:37:22.976867 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:37:22.977449 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:37:22.978231 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:37:22.978277 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:22.978335 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:37:22.978394 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:22.986783 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42543
I20260619 14:37:22.986935 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42543 every 8 connection(s)
I20260619 14:37:22.987702 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:37:22.994839 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 1292
I20260619 14:37:22.994966 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781879837648036-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:37:22.995273 392 external_mini_cluster.cc:1428] Setting key 77b91331dc595431a9e0aa30ce6daa44
Jun 19 14:37:22 dist-test-slave-32hj krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879842, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:37:23.002555 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:22.989625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53659 (local address 127.0.98.62:46301)
0619 14:37:22.989874 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:22.989878 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:22.991148 (+ 1270us) server_negotiation.cc:407] Connection header received
0619 14:37:22.992020 (+ 872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:22.992023 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:22.992073 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:22.992167 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:22.993548 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:22.994410 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:22.995994 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:22.996339 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:22.996435 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:22.999562 (+ 3127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:22.999565 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:22.999567 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:22.999592 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:23.000835 (+ 1243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:23.001312 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:23.001314 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:23.001316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:23.001391 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:23.001636 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:23.001637 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:23.001638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:23.001802 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:23.001882 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:23.002320 (+ 438us) server_negotiation.cc:299] Negotiation successful
0619 14:37:23.002424 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":79,"threads_started":1}
I20260619 14:37:23.003388 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:22.990078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46301 (local address 127.0.98.3:53659)
0619 14:37:22.990963 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:22.991022 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:22.991833 (+ 811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:22.992326 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:22.992333 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:22.992702 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:22.993372 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:22.993396 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:22.994530 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:22.994534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:22.995853 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:22.995864 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:22.996607 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:22.996611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:22.996849 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:22.997513 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:22.997533 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:22.999402 (+ 1869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:23.000957 (+ 1555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:23.000962 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:23.000975 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:23.001238 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:23.001449 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:23.001451 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:23.001452 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:23.001576 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:23.001883 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:23.001888 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:23.002134 (+ 246us) client_negotiation.cc:769] Sending connection context
0619 14:37:23.002290 (+ 156us) client_negotiation.cc:240] Negotiation successful
0619 14:37:23.002546 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":493,"threads_started":1}
I20260619 14:37:23.004451 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46301
I20260619 14:37:23.004671 1421 heartbeater.cc:464] Registering TS with master...
I20260619 14:37:23.005245 1421 heartbeater.cc:511] Master 127.0.98.62:46301 requested a full tablet report, sending...
I20260619 14:37:23.006217 962 ts_manager.cc:194] Registered new tserver with Master: a5528a1f1a39481cb005a49cceeecf39 (127.0.98.3:42543)
I20260619 14:37:23.006752 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53659
I20260619 14:37:23.009859 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:37:23.019838 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:23.012998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39596 (local address 127.0.98.62:46301)
0619 14:37:23.013136 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:23.013140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:23.013375 (+ 235us) server_negotiation.cc:407] Connection header received
0619 14:37:23.013446 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:23.013448 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:23.013486 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:23.013554 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:23.014267 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:23.014798 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:23.015684 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:23.015920 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:23.016005 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:23.017208 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:23.017211 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:23.017213 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:23.017235 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:23.018397 (+ 1162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:23.018885 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:23.018888 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:23.018889 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:23.018961 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:23.019162 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:23.019164 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:23.019166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:23.019366 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:23.019464 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:23.019620 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 14:37:23.019723 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 14:37:23.024293 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39596:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:37:23.027263 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:37:23.040416 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:23.034926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41373 (local address 127.0.0.1:40778)
0619 14:37:23.035308 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:23.035354 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:23.035468 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:23.036011 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:23.036017 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:23.036041 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:37:23.036525 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:23.036532 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:23.037704 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:23.037707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:23.038937 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:23.038947 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:23.039921 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:23.039924 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:23.040036 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:23.040072 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 14:37:23.040166 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 14:37:23.040277 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":92,"threads_started":1}
I20260619 14:37:23.040436 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:23.035087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40778 (local address 127.0.98.1:41373)
0619 14:37:23.035431 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:23.035437 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:23.035462 (+ 25us) server_negotiation.cc:407] Connection header received
0619 14:37:23.035621 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:23.035626 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:23.035847 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:23.036023 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:37:23.036642 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:23.037614 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:23.039068 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:23.039812 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:23.039905 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:23.040044 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:23.040214 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 14:37:23.040308 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":78,"threads_started":1}
I20260619 14:37:23.040939 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:23.035311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43969 (local address 127.0.0.1:59560)
0619 14:37:23.036198 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:23.036211 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:23.036289 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:23.036761 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:23.036764 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:23.036780 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:37:23.036858 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:23.036864 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:23.038025 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:23.038028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:23.039490 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:23.039500 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:23.040621 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:23.040624 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:23.040698 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:23.040738 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 14:37:23.040795 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 14:37:23.040844 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":833,"thread_start_us":104,"threads_started":1}
I20260619 14:37:23.041010 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:23.035831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55148 (local address 127.0.98.3:42543)
0619 14:37:23.036567 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:23.036575 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:23.036594 (+ 19us) server_negotiation.cc:407] Connection header received
0619 14:37:23.036658 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:23.036663 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:23.036864 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:23.036996 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:37:23.037349 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:23.038276 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:23.039452 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:23.040211 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:23.040310 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:23.040438 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:23.040775 (+ 337us) server_negotiation.cc:299] Negotiation successful
0619 14:37:23.040875 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":141,"threads_started":1}
I20260619 14:37:23.040938 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:23.035626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42543 (local address 127.0.0.1:55148)
0619 14:37:23.036295 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:23.036306 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:23.036368 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:23.037137 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:23.037140 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:23.037156 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:37:23.037235 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:23.037240 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:23.038385 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:23.038388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:23.039348 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:23.039357 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:23.040500 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:23.040504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:23.040598 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:23.040619 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 14:37:23.040701 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 14:37:23.040803 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":74,"threads_started":1}
I20260619 14:37:23.042004 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:23.035692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59560 (local address 127.0.98.2:43969)
0619 14:37:23.036313 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:23.036320 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:23.036357 (+ 37us) server_negotiation.cc:407] Connection header received
0619 14:37:23.036427 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:23.036432 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:23.036624 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:23.036769 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:37:23.036954 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:23.037910 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:23.039601 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:23.040354 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:23.040437 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:23.041784 (+ 1347us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:23.041830 (+ 46us) server_negotiation.cc:299] Negotiation successful
0619 14:37:23.041900 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":77,"threads_started":1}
I20260619 14:37:23.043606 1087 tablet_service.cc:1467] Processing CreateTablet for tablet b6ce82ba24254d0ab86547cdc163c51c (DEFAULT_TABLE table=test-table [id=e88c14f45fd546e7ae88f9a7cd936395]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:37:23.043830 1355 tablet_service.cc:1467] Processing CreateTablet for tablet b6ce82ba24254d0ab86547cdc163c51c (DEFAULT_TABLE table=test-table [id=e88c14f45fd546e7ae88f9a7cd936395]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:37:23.044696 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6ce82ba24254d0ab86547cdc163c51c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:23.044744 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6ce82ba24254d0ab86547cdc163c51c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:23.044709 1221 tablet_service.cc:1467] Processing CreateTablet for tablet b6ce82ba24254d0ab86547cdc163c51c (DEFAULT_TABLE table=test-table [id=e88c14f45fd546e7ae88f9a7cd936395]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:37:23.045642 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6ce82ba24254d0ab86547cdc163c51c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:23.050644 1440 tablet_bootstrap.cc:492] T b6ce82ba24254d0ab86547cdc163c51c P a5528a1f1a39481cb005a49cceeecf39: Bootstrap starting.
I20260619 14:37:23.052070 1442 tablet_bootstrap.cc:492] T b6ce82ba24254d0ab86547cdc163c51c P 6e9ba308299f41d6bf99baf56f8df661: Bootstrap starting.
I20260619 14:37:23.052640 1441 tablet_bootstrap.cc:492] T b6ce82ba24254d0ab86547cdc163c51c P ee0b1ab1b4de4808b72c4d5949f92f4a: Bootstrap starting.
I20260619 14:37:23.053162 1440 tablet_bootstrap.cc:654] T b6ce82ba24254d0ab86547cdc163c51c P a5528a1f1a39481cb005a49cceeecf39: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:23.054311 1442 tablet_bootstrap.cc:654] T b6ce82ba24254d0ab86547cdc163c51c P 6e9ba308299f41d6bf99baf56f8df661: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:23.054502 1441 tablet_bootstrap.cc:654] T b6ce82ba24254d0ab86547cdc163c51c P ee0b1ab1b4de4808b72c4d5949f92f4a: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:23.054647 1440 log.cc:824] T b6ce82ba24254d0ab86547cdc163c51c P a5528a1f1a39481cb005a49cceeecf39: Log is configured to *not* fsync() on all Append() calls
I20260619 14:37:23.055078 1442 log.cc:824] T b6ce82ba24254d0ab86547cdc163c51c P 6e9ba308299f41d6bf99baf56f8df661: Log is configured to *not* fsync() on all Append() calls
I20260619 14:37:23.055259 1441 log.cc:824] T b6ce82ba24254d0ab86547cdc163c51c P ee0b1ab1b4de4808b72c4d5949f92f4a: Log is configured to *not* fsync() on all Append() calls
I20260619 14:37:23.056631 1440 tablet_bootstrap.cc:492] T b6ce82ba24254d0ab86547cdc163c51c P a5528a1f1a39481cb005a49cceeecf39: No bootstrap required, opened a new log
I20260619 14:37:23.056830 1440 ts_tablet_manager.cc:1397] T b6ce82ba24254d0ab86547cdc163c51c P a5528a1f1a39481cb005a49cceeecf39: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260619 14:37:23.057020 1441 tablet_bootstrap.cc:492] T b6ce82ba24254d0ab86547cdc163c51c P ee0b1ab1b4de4808b72c4d5949f92f4a: No bootstrap required, opened a new log
I20260619 14:37:23.057179 1441 ts_tablet_manager.cc:1397] T b6ce82ba24254d0ab86547cdc163c51c P ee0b1ab1b4de4808b72c4d5949f92f4a: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260619 14:37:23.060299 1440 raft_consensus.cc:348] T b6ce82ba24254d0ab86547cdc163c51c P a5528a1f1a39481cb005a49cceeecf39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0b1ab1b4de4808b72c4d5949f92f4a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41373 } } peers { permanent_uuid: "a5528a1f1a39481cb005a49cceeecf39" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42543 } } peers { permanent_uuid: "6e9ba308299f41d6bf99baf56f8df661" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43969 } }
I20260619 14:37:23.060588 1440 raft_consensus.cc:374] T b6ce82ba24254d0ab86547cdc163c51c P a5528a1f1a39481cb005a49cceeecf39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:23.060648 1440 raft_consensus.cc:729] T b6ce82ba24254d0ab86547cdc163c51c P a5528a1f1a39481cb005a49cceeecf39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5528a1f1a39481cb005a49cceeecf39, State: Initialized, Role: FOLLOWER
I20260619 14:37:23.060523 1441 raft_consensus.cc:348] T b6ce82ba24254d0ab86547cdc163c51c P ee0b1ab1b4de4808b72c4d5949f92f4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0b1ab1b4de4808b72c4d5949f92f4a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41373 } } peers { permanent_uuid: "a5528a1f1a39481cb005a49cceeecf39" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42543 } } peers { permanent_uuid: "6e9ba308299f41d6bf99baf56f8df661" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43969 } }
I20260619 14:37:23.060786 1441 raft_consensus.cc:374] T b6ce82ba24254d0ab86547cdc163c51c P ee0b1ab1b4de4808b72c4d5949f92f4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:23.060847 1441 raft_consensus.cc:729] T b6ce82ba24254d0ab86547cdc163c51c P ee0b1ab1b4de4808b72c4d5949f92f4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0b1ab1b4de4808b72c4d5949f92f4a, State: Initialized, Role: FOLLOWER
I20260619 14:37:23.061138 1440 consensus_queue.cc:260] T b6ce82ba24254d0ab86547cdc163c51c P a5528a1f1a39481cb005a49cceeecf39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0b1ab1b4de4808b72c4d5949f92f4a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41373 } } peers { permanent_uuid: "a5528a1f1a39481cb005a49cceeecf39" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42543 } } peers { permanent_uuid: "6e9ba308299f41d6bf99baf56f8df661" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43969 } }
I20260619 14:37:23.061241 1441 consensus_queue.cc:260] T b6ce82ba24254d0ab86547cdc163c51c P ee0b1ab1b4de4808b72c4d5949f92f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0b1ab1b4de4808b72c4d5949f92f4a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41373 } } peers { permanent_uuid: "a5528a1f1a39481cb005a49cceeecf39" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42543 } } peers { permanent_uuid: "6e9ba308299f41d6bf99baf56f8df661" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43969 } }
I20260619 14:37:23.061785 1421 heartbeater.cc:503] Master 127.0.98.62:46301 was elected leader, sending a full tablet report...
I20260619 14:37:23.062943 1440 ts_tablet_manager.cc:1428] T b6ce82ba24254d0ab86547cdc163c51c P a5528a1f1a39481cb005a49cceeecf39: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 14:37:23.063742 1441 ts_tablet_manager.cc:1428] T b6ce82ba24254d0ab86547cdc163c51c P ee0b1ab1b4de4808b72c4d5949f92f4a: Time spent starting tablet: real 0.007s user 0.003s sys 0.002s
I20260619 14:37:23.064800 1442 tablet_bootstrap.cc:492] T b6ce82ba24254d0ab86547cdc163c51c P 6e9ba308299f41d6bf99baf56f8df661: No bootstrap required, opened a new log
I20260619 14:37:23.065013 1442 ts_tablet_manager.cc:1397] T b6ce82ba24254d0ab86547cdc163c51c P 6e9ba308299f41d6bf99baf56f8df661: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20260619 14:37:23.068562 1442 raft_consensus.cc:348] T b6ce82ba24254d0ab86547cdc163c51c P 6e9ba308299f41d6bf99baf56f8df661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0b1ab1b4de4808b72c4d5949f92f4a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41373 } } peers { permanent_uuid: "a5528a1f1a39481cb005a49cceeecf39" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42543 } } peers { permanent_uuid: "6e9ba308299f41d6bf99baf56f8df661" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43969 } }
I20260619 14:37:23.068797 1442 raft_consensus.cc:374] T b6ce82ba24254d0ab86547cdc163c51c P 6e9ba308299f41d6bf99baf56f8df661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:23.068857 1442 raft_consensus.cc:729] T b6ce82ba24254d0ab86547cdc163c51c P 6e9ba308299f41d6bf99baf56f8df661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e9ba308299f41d6bf99baf56f8df661, State: Initialized, Role: FOLLOWER
I20260619 14:37:23.069231 1442 consensus_queue.cc:260] T b6ce82ba24254d0ab86547cdc163c51c P 6e9ba308299f41d6bf99baf56f8df661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0b1ab1b4de4808b72c4d5949f92f4a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41373 } } peers { permanent_uuid: "a5528a1f1a39481cb005a49cceeecf39" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42543 } } peers { permanent_uuid: "6e9ba308299f41d6bf99baf56f8df661" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43969 } }
I20260619 14:37:23.070127 1442 ts_tablet_manager.cc:1428] T b6ce82ba24254d0ab86547cdc163c51c P 6e9ba308299f41d6bf99baf56f8df661: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 14:37:23.125862 1448 raft_consensus.cc:482] T b6ce82ba24254d0ab86547cdc163c51c P 6e9ba308299f41d6bf99baf56f8df661 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:37:23.126113 1448 raft_consensus.cc:504] T b6ce82ba24254d0ab86547cdc163c51c P 6e9ba308299f41d6bf99baf56f8df661 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0b1ab1b4de4808b72c4d5949f92f4a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41373 } } peers { permanent_uuid: "a5528a1f1a39481cb005a49cceeecf39" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42543 } } peers { permanent_uuid: "6e9ba308299f41d6bf99baf56f8df661" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43969 } }
I20260619 14:37:23.127365 1448 leader_election.cc:288] T b6ce82ba24254d0ab86547cdc163c51c P 6e9ba308299f41d6bf99baf56f8df661 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee0b1ab1b4de4808b72c4d5949f92f4a (127.0.98.1:41373), a5528a1f1a39481cb005a49cceeecf39 (127.0.98.3:42543)
I20260619 14:37:23.131438 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:23.127691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44813 (local address 127.0.98.1:41373)
0619 14:37:23.127821 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:23.127825 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:23.128008 (+ 183us) server_negotiation.cc:407] Connection header received
0619 14:37:23.128160 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:23.128163 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:23.128220 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:23.128337 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:37:23.128641 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:23.129251 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:23.130394 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:23.130832 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:23.130917 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:23.130981 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:23.131216 (+ 235us) server_negotiation.cc:299] Negotiation successful
0619 14:37:23.131297 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 14:37:23.131438 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:23.128147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33173 (local address 127.0.98.3:42543)
0619 14:37:23.128279 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:23.128284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:23.128421 (+ 137us) server_negotiation.cc:407] Connection header received
0619 14:37:23.128540 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:23.128543 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:23.128587 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:23.128656 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:37:23.128924 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:23.129420 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:23.130457 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:23.130868 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:23.130972 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:23.131030 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:23.131237 (+ 207us) server_negotiation.cc:299] Negotiation successful
0619 14:37:23.131309 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 14:37:23.131464 1450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:23.128016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42543 (local address 127.0.98.2:33173)
0619 14:37:23.128339 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:23.128353 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:23.128465 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:23.128662 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:23.128664 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:23.128680 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:37:23.128828 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:23.128835 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:23.129538 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:23.129541 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:23.130320 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:23.130330 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:23.130982 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:23.130985 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:23.131075 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:23.131140 (+ 65us) client_negotiation.cc:769] Sending connection context
0619 14:37:23.131208 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 14:37:23.131326 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":56,"spinlock_wait_cycles":768,"thread_start_us":95,"threads_started":1}
I20260619 14:37:23.131481 1449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:23.127617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41373 (local address 127.0.98.2:44813)
0619 14:37:23.127917 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:23.127932 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:23.128042 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:23.128358 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:23.128361 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:23.128380 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:37:23.128531 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:23.128538 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:23.129360 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:23.129363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:23.130235 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:23.130244 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:23.130961 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:23.130965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:23.131060 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:23.131126 (+ 66us) client_negotiation.cc:769] Sending connection context
0619 14:37:23.131183 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 14:37:23.131340 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":55,"spinlock_wait_cycles":21120,"thread_start_us":100,"threads_started":1}
I20260619 14:37:23.132043 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6ce82ba24254d0ab86547cdc163c51c" candidate_uuid: "6e9ba308299f41d6bf99baf56f8df661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0b1ab1b4de4808b72c4d5949f92f4a" is_pre_election: true
I20260619 14:37:23.132184 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6ce82ba24254d0ab86547cdc163c51c" candidate_uuid: "6e9ba308299f41d6bf99baf56f8df661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5528a1f1a39481cb005a49cceeecf39" is_pre_election: true
I20260619 14:37:23.132339 1107 raft_consensus.cc:2454] T b6ce82ba24254d0ab86547cdc163c51c P ee0b1ab1b4de4808b72c4d5949f92f4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e9ba308299f41d6bf99baf56f8df661 in term 0.
I20260619 14:37:23.132465 1375 raft_consensus.cc:2454] T b6ce82ba24254d0ab86547cdc163c51c P a5528a1f1a39481cb005a49cceeecf39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e9ba308299f41d6bf99baf56f8df661 in term 0.
I20260619 14:37:23.132817 1177 leader_election.cc:302] T b6ce82ba24254d0ab86547cdc163c51c P 6e9ba308299f41d6bf99baf56f8df661 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e9ba308299f41d6bf99baf56f8df661, ee0b1ab1b4de4808b72c4d5949f92f4a; no voters:
I20260619 14:37:23.133062 1448 raft_consensus.cc:2781] T b6ce82ba24254d0ab86547cdc163c51c P 6e9ba308299f41d6bf99baf56f8df661 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:37:23.133145 1448 raft_consensus.cc:482] T b6ce82ba24254d0ab86547cdc163c51c P 6e9ba308299f41d6bf99baf56f8df661 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:37:23.133193 1448 raft_consensus.cc:3037] T b6ce82ba24254d0ab86547cdc163c51c P 6e9ba308299f41d6bf99baf56f8df661 [term 0 FOLLOWER]: Advancing to term 1
W20260619 14:37:23.140805 1288 tablet.cc:2369] T b6ce82ba24254d0ab86547cdc163c51c P 6e9ba308299f41d6bf99baf56f8df661: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:37:23.188377 1448 raft_consensus.cc:504] T b6ce82ba24254d0ab86547cdc163c51c P 6e9ba308299f41d6bf99baf56f8df661 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0b1ab1b4de4808b72c4d5949f92f4a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41373 } } peers { permanent_uuid: "a5528a1f1a39481cb005a49cceeecf39" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42543 } } peers { permanent_uuid: "6e9ba308299f41d6bf99baf56f8df661" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43969 } }
I20260619 14:37:23.188843 1448 leader_election.cc:288] T b6ce82ba24254d0ab86547cdc163c51c P 6e9ba308299f41d6bf99baf56f8df661 [CANDIDATE]: Term 1 election: Requested vote from peers ee0b1ab1b4de4808b72c4d5949f92f4a (127.0.98.1:41373), a5528a1f1a39481cb005a49cceeecf39 (127.0.98.3:42543)
I20260619 14:37:23.189246 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6ce82ba24254d0ab86547cdc163c51c" candidate_uuid: "6e9ba308299f41d6bf99baf56f8df661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0b1ab1b4de4808b72c4d5949f92f4a"
I20260619 14:37:23.189337 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6ce82ba24254d0ab86547cdc163c51c" candidate_uuid: "6e9ba308299f41d6bf99baf56f8df661" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5528a1f1a39481cb005a49cceeecf39"
I20260619 14:37:23.189401 1107 raft_consensus.cc:3037] T b6ce82ba24254d0ab86547cdc163c51c P ee0b1ab1b4de4808b72c4d5949f92f4a [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:23.189464 1375 raft_consensus.cc:3037] T b6ce82ba24254d0ab86547cdc163c51c P a5528a1f1a39481cb005a49cceeecf39 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:23.190750 1107 raft_consensus.cc:2454] T b6ce82ba24254d0ab86547cdc163c51c P ee0b1ab1b4de4808b72c4d5949f92f4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e9ba308299f41d6bf99baf56f8df661 in term 1.
I20260619 14:37:23.190750 1375 raft_consensus.cc:2454] T b6ce82ba24254d0ab86547cdc163c51c P a5528a1f1a39481cb005a49cceeecf39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e9ba308299f41d6bf99baf56f8df661 in term 1.
I20260619 14:37:23.191107 1177 leader_election.cc:302] T b6ce82ba24254d0ab86547cdc163c51c P 6e9ba308299f41d6bf99baf56f8df661 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e9ba308299f41d6bf99baf56f8df661, a5528a1f1a39481cb005a49cceeecf39; no voters:
I20260619 14:37:23.191279 1448 raft_consensus.cc:2781] T b6ce82ba24254d0ab86547cdc163c51c P 6e9ba308299f41d6bf99baf56f8df661 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:37:23.191501 1448 raft_consensus.cc:686] T b6ce82ba24254d0ab86547cdc163c51c P 6e9ba308299f41d6bf99baf56f8df661 [term 1 LEADER]: Becoming Leader. State: Replica: 6e9ba308299f41d6bf99baf56f8df661, State: Running, Role: LEADER
I20260619 14:37:23.191833 1448 consensus_queue.cc:237] T b6ce82ba24254d0ab86547cdc163c51c P 6e9ba308299f41d6bf99baf56f8df661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0b1ab1b4de4808b72c4d5949f92f4a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41373 } } peers { permanent_uuid: "a5528a1f1a39481cb005a49cceeecf39" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42543 } } peers { permanent_uuid: "6e9ba308299f41d6bf99baf56f8df661" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43969 } }
I20260619 14:37:23.194898 961 catalog_manager.cc:5495] T b6ce82ba24254d0ab86547cdc163c51c P 6e9ba308299f41d6bf99baf56f8df661 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e9ba308299f41d6bf99baf56f8df661 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "6e9ba308299f41d6bf99baf56f8df661" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0b1ab1b4de4808b72c4d5949f92f4a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5528a1f1a39481cb005a49cceeecf39" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e9ba308299f41d6bf99baf56f8df661" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43969 } health_report { overall_health: HEALTHY } } }
W20260619 14:37:23.196518 1154 tablet.cc:2369] T b6ce82ba24254d0ab86547cdc163c51c P ee0b1ab1b4de4808b72c4d5949f92f4a: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 14:37:23 dist-test-slave-32hj krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 14:37:23.233857 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:23.225817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40788 (local address 127.0.98.1:41373)
0619 14:37:23.225943 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:23.225948 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:23.226106 (+ 158us) server_negotiation.cc:407] Connection header received
0619 14:37:23.226231 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:23.226234 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:23.226283 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:23.226349 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:23.227002 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:23.227497 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:23.228284 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:23.228497 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:23.228581 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:23.230844 (+ 2263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:23.230853 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:23.230868 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:23.230890 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:23.232231 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:23.232609 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:23.232615 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:23.232616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:23.232694 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:23.232969 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:23.232972 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:23.232974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:23.233209 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:23.233403 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:23.233581 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 14:37:23.233692 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260619 14:37:23.238615 1422 tablet.cc:2369] T b6ce82ba24254d0ab86547cdc163c51c P a5528a1f1a39481cb005a49cceeecf39: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 14:37:23 dist-test-slave-32hj krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879843, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 14:37:23 dist-test-slave-32hj krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879843, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 14:37:23.291383 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:23.282350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40800 (local address 127.0.98.1:41373)
0619 14:37:23.282498 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:23.282503 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:23.282668 (+ 165us) server_negotiation.cc:407] Connection header received
0619 14:37:23.283486 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:23.283489 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:23.283534 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:23.283604 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:23.284429 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:23.284986 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:23.285869 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:23.286137 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:23.286336 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:23.288687 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:23.288690 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:23.288691 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:23.288744 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:23.289914 (+ 1170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:23.290322 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:23.290324 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:23.290325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:23.290400 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:23.290583 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:23.290585 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:23.290586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:23.290735 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:23.290858 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:23.291111 (+ 253us) server_negotiation.cc:299] Negotiation successful
0619 14:37:23.291222 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Jun 19 14:37:23 dist-test-slave-32hj krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 14:37:23 dist-test-slave-32hj krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 14:37:23.392108 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:23.384107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40806 (local address 127.0.98.1:41373)
0619 14:37:23.384249 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:23.384254 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:23.384443 (+ 189us) server_negotiation.cc:407] Connection header received
0619 14:37:23.384597 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:23.384599 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:23.384658 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:23.384779 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:23.385594 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:23.386123 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:23.386929 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:23.387140 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:23.387220 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:23.389512 (+ 2292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:23.389516 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:23.389518 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:23.389541 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:23.390645 (+ 1104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:23.390994 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:23.390997 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:23.390998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:23.391063 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:23.391273 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:23.391276 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:23.391277 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:23.391462 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:23.391600 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:23.391841 (+ 241us) server_negotiation.cc:299] Negotiation successful
0619 14:37:23.391961 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260619 14:37:23.393617 1067 tablet_service.cc:2908] Rejecting scan request for tablet b6ce82ba24254d0ab86547cdc163c51c: Uninitialized: clean time has not yet been initialized
I20260619 14:37:23.395457 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 1024
I20260619 14:37:23.400588 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 1158
I20260619 14:37:23.405840 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 1292
I20260619 14:37:23.410220 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 930
2026-06-19T14:37:23Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3705 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:37:24 dist-test-slave-32hj krb5kdc[1465](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:37:24 dist-test-slave-32hj krb5kdc[1465](info): set up 2 sockets
Jun 19 14:37:24 dist-test-slave-32hj krb5kdc[1465](info): commencing operation
krb5kdc: starting...
W20260619 14:37:25.520201 392 mini_kdc.cc:121] Time spent starting KDC: real 2.090s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260619 14:37:25.715523 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.195s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260619 14:37:25.919644 392 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.204s user 0.000s sys 0.001s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/krb5kdc/test-user.keytab.
W20260619 14:37:26.913494 392 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.915s user 0.000s sys 0.001s
Jun 19 14:37:26 dist-test-slave-32hj krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:37:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:37:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:37:26.956459 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43579
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43579
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:37:27.063719 1478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:37:27.063971 1478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:37:27.064009 1478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:37:27.068029 1478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:37:27.068085 1478 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 14:37:27.068100 1478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:37:27.068117 1478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:37:27.068131 1478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:37:27.068150 1478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:37:27.073335 1478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43579
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:37:27.074316 1478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:37:27.075306 1478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:37:27.083469 1483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:27.083542 1484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:27.083587 1486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:27.083518 1478 server_base.cc:1034] running on GCE node
I20260619 14:37:27.084193 1478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:37:27.085335 1478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:37:27.086547 1478 hybrid_clock.cc:648] HybridClock initialized: now 1781879847086525 us; error 37 us; skew 500 ppm
Jun 19 14:37:27 dist-test-slave-32hj krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879847, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:37:27.089213 1478 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:37:27.090008 1478 webserver.cc:460] Webserver started at http://127.0.98.62:38437/ using document root <none> and password file <none>
I20260619 14:37:27.090595 1478 fs_manager.cc:359] Metadata directory not provided
I20260619 14:37:27.090665 1478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:37:27.090842 1478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:37:27.092808 1478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/master-0/data/instance:
uuid: "82fc67ef30144ceabdac56aab5b87dc8"
format_stamp: "Formatted at 2026-06-19 14:37:27 on dist-test-slave-32hj"
server_key: "73eb0305c5a2138efc010576e063b8af"
server_key_iv: "7795ee0d32935298dc25de3971e80500"
server_key_version: "encryptionkey@0"
I20260619 14:37:27.093307 1478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/master-0/wal/instance:
uuid: "82fc67ef30144ceabdac56aab5b87dc8"
format_stamp: "Formatted at 2026-06-19 14:37:27 on dist-test-slave-32hj"
server_key: "73eb0305c5a2138efc010576e063b8af"
server_key_iv: "7795ee0d32935298dc25de3971e80500"
server_key_version: "encryptionkey@0"
I20260619 14:37:27.096689 1478 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260619 14:37:27.098970 1493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:27.100128 1478 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:37:27.100246 1478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/master-0/wal
uuid: "82fc67ef30144ceabdac56aab5b87dc8"
format_stamp: "Formatted at 2026-06-19 14:37:27 on dist-test-slave-32hj"
server_key: "73eb0305c5a2138efc010576e063b8af"
server_key_iv: "7795ee0d32935298dc25de3971e80500"
server_key_version: "encryptionkey@0"
I20260619 14:37:27.100353 1478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:37:27.144330 1478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:37:27.147590 1478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:37:27.147776 1478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:37:27.155450 1478 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43579
I20260619 14:37:27.155507 1555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43579 every 8 connection(s)
I20260619 14:37:27.156404 1478 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:37:27.159080 1556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:27.162062 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 1478
I20260619 14:37:27.162184 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/master-0/wal/instance
I20260619 14:37:27.162429 392 external_mini_cluster.cc:1428] Setting key 59c1292fef8839a4d62b2f5cca499285
I20260619 14:37:27.164875 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82fc67ef30144ceabdac56aab5b87dc8: Bootstrap starting.
I20260619 14:37:27.167223 1556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82fc67ef30144ceabdac56aab5b87dc8: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:27.167918 1556 log.cc:824] T 00000000000000000000000000000000 P 82fc67ef30144ceabdac56aab5b87dc8: Log is configured to *not* fsync() on all Append() calls
I20260619 14:37:27.169811 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82fc67ef30144ceabdac56aab5b87dc8: No bootstrap required, opened a new log
Jun 19 14:37:27 dist-test-slave-32hj krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:37:27.173231 1556 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82fc67ef30144ceabdac56aab5b87dc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82fc67ef30144ceabdac56aab5b87dc8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43579 } }
I20260619 14:37:27.173455 1556 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82fc67ef30144ceabdac56aab5b87dc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:27.173518 1556 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82fc67ef30144ceabdac56aab5b87dc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82fc67ef30144ceabdac56aab5b87dc8, State: Initialized, Role: FOLLOWER
I20260619 14:37:27.173945 1556 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82fc67ef30144ceabdac56aab5b87dc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82fc67ef30144ceabdac56aab5b87dc8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43579 } }
I20260619 14:37:27.174072 1556 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82fc67ef30144ceabdac56aab5b87dc8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:37:27.174155 1556 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82fc67ef30144ceabdac56aab5b87dc8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:37:27.174258 1556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82fc67ef30144ceabdac56aab5b87dc8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:27.175161 1556 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82fc67ef30144ceabdac56aab5b87dc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82fc67ef30144ceabdac56aab5b87dc8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43579 } }
I20260619 14:37:27.175472 1556 leader_election.cc:302] T 00000000000000000000000000000000 P 82fc67ef30144ceabdac56aab5b87dc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82fc67ef30144ceabdac56aab5b87dc8; no voters:
I20260619 14:37:27.175534 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:27.163722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49552 (local address 127.0.98.62:43579)
0619 14:37:27.164108 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:27.164124 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:27.164158 (+ 34us) server_negotiation.cc:407] Connection header received
0619 14:37:27.164784 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:27.164800 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:27.165152 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:27.165434 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:27.166262 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:27.167239 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:27.168140 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:27.168812 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:27.168952 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:27.171208 (+ 2256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:27.171213 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:27.171226 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:27.171251 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:27.172881 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:27.173278 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:27.173284 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:27.173290 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:27.173400 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:27.173654 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:27.173657 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:27.173659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:27.174039 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:27.174228 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:27.174491 (+ 263us) server_negotiation.cc:299] Negotiation successful
0619 14:37:27.174768 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":106,"threads_started":1}
I20260619 14:37:27.175786 1556 leader_election.cc:288] T 00000000000000000000000000000000 P 82fc67ef30144ceabdac56aab5b87dc8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:37:27.175870 1561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82fc67ef30144ceabdac56aab5b87dc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:37:27.176038 1561 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82fc67ef30144ceabdac56aab5b87dc8 [term 1 LEADER]: Becoming Leader. State: Replica: 82fc67ef30144ceabdac56aab5b87dc8, State: Running, Role: LEADER
I20260619 14:37:27.176400 1561 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82fc67ef30144ceabdac56aab5b87dc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82fc67ef30144ceabdac56aab5b87dc8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43579 } }
I20260619 14:37:27.177019 1556 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82fc67ef30144ceabdac56aab5b87dc8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:37:27.177551 1561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82fc67ef30144ceabdac56aab5b87dc8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82fc67ef30144ceabdac56aab5b87dc8. Latest consensus state: current_term: 1 leader_uuid: "82fc67ef30144ceabdac56aab5b87dc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82fc67ef30144ceabdac56aab5b87dc8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43579 } } }
I20260619 14:37:27.177654 1561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82fc67ef30144ceabdac56aab5b87dc8 [sys.catalog]: This master's current role is: LEADER
I20260619 14:37:27.178203 1562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82fc67ef30144ceabdac56aab5b87dc8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82fc67ef30144ceabdac56aab5b87dc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82fc67ef30144ceabdac56aab5b87dc8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43579 } } }
I20260619 14:37:27.178295 1562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82fc67ef30144ceabdac56aab5b87dc8 [sys.catalog]: This master's current role is: LEADER
W20260619 14:37:27.181299 1576 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 82fc67ef30144ceabdac56aab5b87dc8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:37:27.181360 1576 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:37:27.181440 1566 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:37:27.182082 1566 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:37:27.187203 1566 catalog_manager.cc:1347] Generated new cluster ID: e5f9c49c37764df88df1bd057822086f
I20260619 14:37:27.187258 1566 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:37:27.196022 1566 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:37:27.197189 1566 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:37:27.203823 1566 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82fc67ef30144ceabdac56aab5b87dc8: Generated new TSK 0
I20260619 14:37:27.204562 1566 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 14:37:27.231745 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43579
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:37:27.372357 1582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:37:27.372574 1582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:37:27.372613 1582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:37:27.376431 1582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:37:27.376477 1582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:37:27.376513 1582 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 14:37:27.376547 1582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:37:27.381448 1582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:37:27.382299 1582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:37:27.383176 1582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:37:27.391690 1590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:27.391723 1587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:27.391867 1582 server_base.cc:1034] running on GCE node
W20260619 14:37:27.391708 1588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:27.392269 1582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:37:27.392879 1582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:37:27.394032 1582 hybrid_clock.cc:648] HybridClock initialized: now 1781879847394015 us; error 29 us; skew 500 ppm
Jun 19 14:37:27 dist-test-slave-32hj krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879847, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:37:27.396420 1582 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:37:27.397166 1582 webserver.cc:460] Webserver started at http://127.0.98.1:38793/ using document root <none> and password file <none>
I20260619 14:37:27.397621 1582 fs_manager.cc:359] Metadata directory not provided
I20260619 14:37:27.397673 1582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:37:27.397821 1582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:37:27.399610 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/ts-0/data/instance:
uuid: "422fcf83e69b42988c71e1a5a7774760"
format_stamp: "Formatted at 2026-06-19 14:37:27 on dist-test-slave-32hj"
server_key: "c8937ed93a997d79ad4aa9df8daac8b1"
server_key_iv: "88b5e42bdaf47a16075421f1e63080b0"
server_key_version: "encryptionkey@0"
I20260619 14:37:27.400086 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/ts-0/wal/instance:
uuid: "422fcf83e69b42988c71e1a5a7774760"
format_stamp: "Formatted at 2026-06-19 14:37:27 on dist-test-slave-32hj"
server_key: "c8937ed93a997d79ad4aa9df8daac8b1"
server_key_iv: "88b5e42bdaf47a16075421f1e63080b0"
server_key_version: "encryptionkey@0"
I20260619 14:37:27.403367 1582 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:37:27.405637 1597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:27.406733 1582 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:37:27.406834 1582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/ts-0/wal
uuid: "422fcf83e69b42988c71e1a5a7774760"
format_stamp: "Formatted at 2026-06-19 14:37:27 on dist-test-slave-32hj"
server_key: "c8937ed93a997d79ad4aa9df8daac8b1"
server_key_iv: "88b5e42bdaf47a16075421f1e63080b0"
server_key_version: "encryptionkey@0"
I20260619 14:37:27.406930 1582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:37:27.427652 1582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:37:27.430150 1582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:37:27.430305 1582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:37:27.431531 1582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:37:27.431573 1582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:27.431612 1582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:37:27.431639 1582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 14:37:27 dist-test-slave-32hj krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879847, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:37:27.441264 1582 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39197
I20260619 14:37:27.441316 1712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39197 every 8 connection(s)
I20260619 14:37:27.442106 1582 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:37:27.444684 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:27.432273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52665 (local address 127.0.98.62:43579)
0619 14:37:27.432383 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:27.432387 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:27.433374 (+ 987us) server_negotiation.cc:407] Connection header received
0619 14:37:27.434325 (+ 951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:27.434328 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:27.434388 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:27.434445 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:27.435951 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:27.436785 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:27.438455 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:27.438681 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:27.438764 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:27.441504 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:27.441507 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:27.441509 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:27.441534 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:27.443042 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:27.443478 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:27.443480 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:27.443482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:27.443551 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:27.443805 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:27.443806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:27.443807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:27.443960 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:27.444026 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:27.444466 (+ 440us) server_negotiation.cc:299] Negotiation successful
0619 14:37:27.444566 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 14:37:27.445401 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:27.432807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43579 (local address 127.0.98.1:52665)
0619 14:37:27.433236 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:27.433280 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:27.434089 (+ 809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:27.434823 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:27.434833 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:27.435226 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:27.435815 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:27.435834 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:27.436935 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:27.436939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:27.438339 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:27.438350 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:27.438833 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:27.438837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:27.439026 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:27.439588 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:27.439605 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:27.441347 (+ 1742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:27.443154 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:27.443159 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:27.443169 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:27.443400 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:27.443614 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:27.443616 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:27.443618 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:27.443730 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:27.444027 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:27.444032 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:27.444232 (+ 200us) client_negotiation.cc:769] Sending connection context
0619 14:37:27.444380 (+ 148us) client_negotiation.cc:240] Negotiation successful
0619 14:37:27.444601 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":115,"threads_started":1}
I20260619 14:37:27.446488 1713 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43579
I20260619 14:37:27.446619 1713 heartbeater.cc:464] Registering TS with master...
I20260619 14:37:27.447126 1713 heartbeater.cc:511] Master 127.0.98.62:43579 requested a full tablet report, sending...
I20260619 14:37:27.447786 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 1582
I20260619 14:37:27.447897 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:37:27.448155 392 external_mini_cluster.cc:1428] Setting key e2b954f310b35753876083f5a780e29b
I20260619 14:37:27.448801 1510 ts_manager.cc:194] Registered new tserver with Master: 422fcf83e69b42988c71e1a5a7774760 (127.0.98.1:39197)
I20260619 14:37:27.450129 1510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52665
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 14:37:27.457727 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:27.448875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52555 (local address 127.0.98.62:43579)
0619 14:37:27.450722 (+ 1847us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:27.450726 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:27.450738 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:37:27.450769 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:27.450772 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:27.450808 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:27.450886 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:27.451383 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:27.452120 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:27.452932 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:27.453257 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:27.453365 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:27.454405 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:27.454409 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:27.454412 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:27.454439 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:27.455894 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:27.456324 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:27.456328 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:27.456330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:27.456415 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:27.456683 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:27.456686 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:27.456688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:27.456927 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:27.457028 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:27.457432 (+ 404us) server_negotiation.cc:299] Negotiation successful
0619 14:37:27.457562 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1781}
I20260619 14:37:27.458226 1719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:27.448564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43579 (local address 127.0.98.1:52555)
0619 14:37:27.448819 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:27.448834 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:27.448912 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:27.450916 (+ 2004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:27.450919 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:27.451112 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:27.451262 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:27.451270 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:27.452253 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:27.452256 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:27.452834 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:27.452842 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:27.453385 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:27.453388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:27.453461 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:27.453904 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:27.453915 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:27.454271 (+ 356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:27.456004 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:27.456006 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:27.456008 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:27.456232 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:27.456511 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:27.456514 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:27.456515 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:27.456596 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:27.457066 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:27.457069 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:27.457192 (+ 123us) client_negotiation.cc:769] Sending connection context
0619 14:37:27.457873 (+ 681us) client_negotiation.cc:240] Negotiation successful
0619 14:37:27.458053 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":2,"thread_start_us":89,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 14:37:27.471200 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43579
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:37:27.574941 1722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:37:27.575194 1722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:37:27.575237 1722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:37:27.579272 1722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:37:27.579321 1722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:37:27.579358 1722 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 14:37:27.579397 1722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:37:27.584467 1722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:37:27.585475 1722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:37:27.586431 1722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:37:27.595336 1728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:27.595335 1727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:27.595371 1722 server_base.cc:1034] running on GCE node
W20260619 14:37:27.595335 1730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:27.596001 1722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:37:27.596581 1722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:37:27.597733 1722 hybrid_clock.cc:648] HybridClock initialized: now 1781879847597696 us; error 41 us; skew 500 ppm
Jun 19 14:37:27 dist-test-slave-32hj krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879847, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:37:27.600281 1722 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:37:27.601131 1722 webserver.cc:460] Webserver started at http://127.0.98.2:36619/ using document root <none> and password file <none>
I20260619 14:37:27.601642 1722 fs_manager.cc:359] Metadata directory not provided
I20260619 14:37:27.601701 1722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:37:27.601897 1722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:37:27.603859 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/ts-1/data/instance:
uuid: "6b28eaa67a5d4711a16af5366ff2b67f"
format_stamp: "Formatted at 2026-06-19 14:37:27 on dist-test-slave-32hj"
server_key: "3667ffc577fb2d73c7d79e6b07d01cb1"
server_key_iv: "98f7ac5341af26b3cebf774745fa5c3c"
server_key_version: "encryptionkey@0"
I20260619 14:37:27.604368 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6b28eaa67a5d4711a16af5366ff2b67f"
format_stamp: "Formatted at 2026-06-19 14:37:27 on dist-test-slave-32hj"
server_key: "3667ffc577fb2d73c7d79e6b07d01cb1"
server_key_iv: "98f7ac5341af26b3cebf774745fa5c3c"
server_key_version: "encryptionkey@0"
I20260619 14:37:27.607831 1722 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 14:37:27.610101 1737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:27.611295 1722 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:37:27.611423 1722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/ts-1/wal
uuid: "6b28eaa67a5d4711a16af5366ff2b67f"
format_stamp: "Formatted at 2026-06-19 14:37:27 on dist-test-slave-32hj"
server_key: "3667ffc577fb2d73c7d79e6b07d01cb1"
server_key_iv: "98f7ac5341af26b3cebf774745fa5c3c"
server_key_version: "encryptionkey@0"
I20260619 14:37:27.611521 1722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:37:27.621336 1722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:37:27.624146 1722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:37:27.624300 1722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:37:27.625605 1722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:37:27.625650 1722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:27.625707 1722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:37:27.625738 1722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 14:37:27 dist-test-slave-32hj krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879847, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:37:27.635432 1722 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37667
I20260619 14:37:27.635454 1852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37667 every 8 connection(s)
I20260619 14:37:27.636291 1722 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:37:27.638187 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:27.626358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54079 (local address 127.0.98.62:43579)
0619 14:37:27.626468 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:27.626473 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:27.627438 (+ 965us) server_negotiation.cc:407] Connection header received
0619 14:37:27.628435 (+ 997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:27.628439 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:27.628492 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:27.628545 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:27.630137 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:27.630910 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:27.632074 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:27.632306 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:27.632392 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:27.634749 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:27.634753 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:27.634756 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:27.634784 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:27.636281 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:27.636845 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:27.636849 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:27.636850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:27.636942 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:27.637270 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:27.637274 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:27.637275 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:27.637461 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:27.637535 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:27.637969 (+ 434us) server_negotiation.cc:299] Negotiation successful
0619 14:37:27.638069 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 14:37:27.638949 1746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:27.626860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43579 (local address 127.0.98.2:54079)
0619 14:37:27.627304 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:27.627342 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:27.628191 (+ 849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:27.628961 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:27.628969 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:27.629383 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:27.629976 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:27.629995 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:27.631015 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:27.631018 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:27.631967 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:27.631976 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:27.632448 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:27.632452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:27.632603 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:27.633160 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:27.633175 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:27.634587 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:27.636385 (+ 1798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:27.636390 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:27.636403 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:27.636746 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:27.637023 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:27.637026 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:27.637028 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:27.637157 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:27.637538 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:27.637542 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:27.637754 (+ 212us) client_negotiation.cc:769] Sending connection context
0619 14:37:27.637895 (+ 141us) client_negotiation.cc:240] Negotiation successful
0619 14:37:27.638128 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":116,"threads_started":1}
I20260619 14:37:27.639991 1853 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43579
I20260619 14:37:27.640122 1853 heartbeater.cc:464] Registering TS with master...
I20260619 14:37:27.640630 1853 heartbeater.cc:511] Master 127.0.98.62:43579 requested a full tablet report, sending...
I20260619 14:37:27.641724 1509 ts_manager.cc:194] Registered new tserver with Master: 6b28eaa67a5d4711a16af5366ff2b67f (127.0.98.2:37667)
I20260619 14:37:27.642424 1509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54079
I20260619 14:37:27.646454 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 1722
I20260619 14:37:27.646566 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:37:27.646898 392 external_mini_cluster.cc:1428] Setting key 1c4dd5ef5dd10759edfdb4412dfa369b
I20260619 14:37:27.648931 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:27.642157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47469 (local address 127.0.98.62:43579)
0619 14:37:27.642259 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:27.642264 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:27.642424 (+ 160us) server_negotiation.cc:407] Connection header received
0619 14:37:27.642517 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:27.642520 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:27.642560 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:27.642623 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:27.643018 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:27.643508 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:27.644473 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:27.644684 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:27.644826 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:27.645706 (+ 880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:27.645709 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:27.645711 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:27.645728 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:27.647302 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:27.647729 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:27.647732 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:27.647734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:27.647813 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:27.648075 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:27.648078 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:27.648079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:27.648298 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:27.648376 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:27.648680 (+ 304us) server_negotiation.cc:299] Negotiation successful
0619 14:37:27.648826 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 14:37:27.648929 1859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:27.642111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43579 (local address 127.0.98.2:47469)
0619 14:37:27.642352 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:27.642364 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:27.642449 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:27.642644 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:27.642647 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:27.642821 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:27.642913 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:27.642920 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:27.643624 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:27.643627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:27.644361 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:27.644370 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:27.644825 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:27.644828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:27.644892 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:27.645316 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:27.645326 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:27.645600 (+ 274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:27.647417 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:27.647420 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:27.647422 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:27.647615 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:27.647891 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:27.647894 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:27.647896 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:27.647988 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:27.648396 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:27.648399 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:27.648492 (+ 93us) client_negotiation.cc:769] Sending connection context
0619 14:37:27.648598 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 14:37:27.648756 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":100,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 14:37:27.671120 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43579
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:37:27.775192 1862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:37:27.775427 1862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:37:27.775466 1862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:37:27.779371 1862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:37:27.779420 1862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:37:27.779456 1862 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 14:37:27.779492 1862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:37:27.784355 1862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:37:27.785290 1862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:37:27.786198 1862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:37:27.794793 1868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:27.794799 1870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:27.794790 1867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:27.795083 1862 server_base.cc:1034] running on GCE node
I20260619 14:37:27.795495 1862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:37:27.796068 1862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:37:27.797214 1862 hybrid_clock.cc:648] HybridClock initialized: now 1781879847797198 us; error 31 us; skew 500 ppm
Jun 19 14:37:27 dist-test-slave-32hj krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879847, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:37:27.799791 1862 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:37:27.800575 1862 webserver.cc:460] Webserver started at http://127.0.98.3:37123/ using document root <none> and password file <none>
I20260619 14:37:27.801112 1862 fs_manager.cc:359] Metadata directory not provided
I20260619 14:37:27.801177 1862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:37:27.801393 1862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:37:27.803296 1862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/ts-2/data/instance:
uuid: "aee5e9280b8a48b6ada6dd7286f51706"
format_stamp: "Formatted at 2026-06-19 14:37:27 on dist-test-slave-32hj"
server_key: "f3a2d98e0e80c4eefabb064a3e4b7dd1"
server_key_iv: "407bd697f6e155b61cfb714274997bd1"
server_key_version: "encryptionkey@0"
I20260619 14:37:27.803828 1862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/ts-2/wal/instance:
uuid: "aee5e9280b8a48b6ada6dd7286f51706"
format_stamp: "Formatted at 2026-06-19 14:37:27 on dist-test-slave-32hj"
server_key: "f3a2d98e0e80c4eefabb064a3e4b7dd1"
server_key_iv: "407bd697f6e155b61cfb714274997bd1"
server_key_version: "encryptionkey@0"
I20260619 14:37:27.807305 1862 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260619 14:37:27.809471 1877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:27.810622 1862 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260619 14:37:27.810720 1862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/ts-2/wal
uuid: "aee5e9280b8a48b6ada6dd7286f51706"
format_stamp: "Formatted at 2026-06-19 14:37:27 on dist-test-slave-32hj"
server_key: "f3a2d98e0e80c4eefabb064a3e4b7dd1"
server_key_iv: "407bd697f6e155b61cfb714274997bd1"
server_key_version: "encryptionkey@0"
I20260619 14:37:27.810828 1862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:37:27.824570 1862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:37:27.826994 1862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:37:27.827148 1862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:37:27.828403 1862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:37:27.828449 1862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:27.828497 1862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:37:27.828526 1862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 14:37:27 dist-test-slave-32hj krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879847, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:37:27.838207 1862 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42729
I20260619 14:37:27.838258 1992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42729 every 8 connection(s)
I20260619 14:37:27.839068 1862 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:37:27.841014 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:27.829150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55109 (local address 127.0.98.62:43579)
0619 14:37:27.829255 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:27.829260 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:27.830222 (+ 962us) server_negotiation.cc:407] Connection header received
0619 14:37:27.831159 (+ 937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:27.831163 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:27.831217 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:27.831269 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:27.832840 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:27.833413 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:27.834560 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:27.834771 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:27.834859 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:27.837539 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:27.837543 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:27.837546 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:27.837572 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:27.838983 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:27.839506 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:27.839509 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:27.839511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:27.839598 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:27.839987 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:27.839991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:27.839992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:27.840171 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:27.840293 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:27.840784 (+ 491us) server_negotiation.cc:299] Negotiation successful
0619 14:37:27.840898 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 14:37:27.841718 1886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:27.829646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43579 (local address 127.0.98.3:55109)
0619 14:37:27.830090 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:27.830128 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:27.830927 (+ 799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:27.831649 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:27.831658 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:27.832094 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:27.832665 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:27.832684 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:27.833529 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:27.833532 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:27.834444 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:27.834453 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:27.834912 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:27.834915 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:27.835074 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:27.835607 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:27.835625 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:27.837388 (+ 1763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:27.839089 (+ 1701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:27.839094 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:27.839107 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:27.839403 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:27.839705 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:27.839709 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:27.839711 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:27.839867 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:27.840339 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:27.840344 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:27.840542 (+ 198us) client_negotiation.cc:769] Sending connection context
0619 14:37:27.840691 (+ 149us) client_negotiation.cc:240] Negotiation successful
0619 14:37:27.840952 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":119,"threads_started":1}
I20260619 14:37:27.842730 1993 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43579
I20260619 14:37:27.842854 1993 heartbeater.cc:464] Registering TS with master...
I20260619 14:37:27.843279 1993 heartbeater.cc:511] Master 127.0.98.62:43579 requested a full tablet report, sending...
I20260619 14:37:27.844221 1510 ts_manager.cc:194] Registered new tserver with Master: aee5e9280b8a48b6ada6dd7286f51706 (127.0.98.3:42729)
I20260619 14:37:27.844904 1510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55109
I20260619 14:37:27.846278 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 1862
I20260619 14:37:27.846385 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:37:27.846637 392 external_mini_cluster.cc:1428] Setting key d988f3a424aaeec4d0912c60146157fb
I20260619 14:37:27.848408 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:37:27.850864 1999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:27.844519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43579 (local address 127.0.98.3:54277)
0619 14:37:27.844809 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:27.844819 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:27.844897 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:27.845095 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:27.845098 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:27.845292 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:27.845401 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:27.845408 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:27.846087 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:27.846090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:27.846867 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:27.846876 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:27.847286 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:27.847290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:27.847377 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:27.847851 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:27.847862 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:27.848116 (+ 254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:27.849636 (+ 1520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:27.849639 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:27.849641 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:27.849857 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:27.850128 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:27.850130 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:27.850132 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:27.850197 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:27.850525 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:27.850527 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:27.850593 (+ 66us) client_negotiation.cc:769] Sending connection context
0619 14:37:27.850662 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 14:37:27.850749 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":78,"threads_started":1}
I20260619 14:37:27.850965 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:27.844584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54277 (local address 127.0.98.62:43579)
0619 14:37:27.844689 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:27.844693 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:27.844877 (+ 184us) server_negotiation.cc:407] Connection header received
0619 14:37:27.844963 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:27.844966 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:27.845007 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:27.845071 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:27.845503 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:27.845976 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:27.846959 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:27.847189 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:27.847269 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:27.848237 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:27.848240 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:27.848243 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:27.848268 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:27.849537 (+ 1269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:27.849959 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:27.849962 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:27.849964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:27.850049 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:27.850272 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:27.850274 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:27.850276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:27.850453 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:27.850539 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:27.850724 (+ 185us) server_negotiation.cc:299] Negotiation successful
0619 14:37:27.850829 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
Jun 19 14:37:27 dist-test-slave-32hj krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879847, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 14:37:27 dist-test-slave-32hj krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879847, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:37:27.871039 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:27.864205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56294 (local address 127.0.98.62:43579)
0619 14:37:27.864316 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:27.864320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:27.864334 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:37:27.864364 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:27.864366 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:27.864398 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:27.864460 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:27.865112 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:27.865595 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:27.866357 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:27.866561 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:27.866642 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:27.868585 (+ 1943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:27.868588 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:27.868590 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:27.868609 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:27.869870 (+ 1261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:27.870219 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:27.870221 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:27.870222 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:27.870286 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:27.870464 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:27.870466 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:27.870467 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:27.870619 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:27.870710 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:27.870861 (+ 151us) server_negotiation.cc:299] Negotiation successful
0619 14:37:27.870937 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 14:37:27.873512 1510 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56294:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:37:27.876338 1510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:37:27.889568 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:27.884049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48714 (local address 127.0.98.2:37667)
0619 14:37:27.884752 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:27.884763 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:27.884785 (+ 22us) server_negotiation.cc:407] Connection header received
0619 14:37:27.884858 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:27.884864 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:27.885077 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:27.885236 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:37:27.885508 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:27.886630 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:27.888360 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:27.888955 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:27.889051 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:27.889205 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:27.889343 (+ 138us) server_negotiation.cc:299] Negotiation successful
0619 14:37:27.889429 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":98,"threads_started":1}
I20260619 14:37:27.889559 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:27.883890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37667 (local address 127.0.0.1:48714)
0619 14:37:27.884303 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:27.884351 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:27.884470 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:27.885225 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:27.885231 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:27.885280 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:37:27.885408 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:27.885416 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:27.886740 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:27.886743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:27.888220 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:27.888231 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:27.889061 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:27.889065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:27.889163 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:27.889199 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 14:37:27.889308 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 14:37:27.889424 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":54,"thread_start_us":61,"threads_started":1}
I20260619 14:37:27.889899 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:27.883973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42729 (local address 127.0.0.1:59774)
0619 14:37:27.884407 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:27.884419 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:27.884480 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:27.885768 (+ 1288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:27.885772 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:27.885790 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:37:27.885872 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:27.885878 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:27.887058 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:27.887061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:27.888198 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:27.888207 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:27.889440 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:27.889444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:27.889573 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:27.889614 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 14:37:27.889680 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 14:37:27.889753 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"mutex_wait_us":123,"thread_start_us":79,"threads_started":1}
I20260619 14:37:27.890195 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:27.884737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56214 (local address 127.0.98.1:39197)
0619 14:37:27.885118 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:27.885129 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:27.885611 (+ 482us) server_negotiation.cc:407] Connection header received
0619 14:37:27.885707 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:27.885712 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:27.885853 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:27.885967 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:37:27.886169 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:27.887003 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:27.888348 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:27.889134 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:27.889234 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:27.889380 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:27.889965 (+ 585us) server_negotiation.cc:299] Negotiation successful
0619 14:37:27.890061 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":90,"threads_started":1}
I20260619 14:37:27.889899 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:27.884535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39197 (local address 127.0.0.1:56214)
0619 14:37:27.885522 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:27.885535 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:27.885611 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:27.885963 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:27.885966 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:27.885981 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:37:27.886075 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:27.886081 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:27.887124 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:27.887127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:27.888265 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:27.888272 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:27.889440 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:27.889444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:27.889589 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:27.889614 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 14:37:27.889680 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 14:37:27.889753 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":931,"mutex_wait_us":36,"thread_start_us":80,"threads_started":1}
I20260619 14:37:27.890367 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:27.884813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59774 (local address 127.0.98.3:42729)
0619 14:37:27.885177 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:27.885187 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:27.885238 (+ 51us) server_negotiation.cc:407] Connection header received
0619 14:37:27.885306 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:27.885311 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:27.885503 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:27.885637 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:37:27.885991 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:27.886962 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:27.888318 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:27.889134 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:27.889234 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:27.889380 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:27.890151 (+ 771us) server_negotiation.cc:299] Negotiation successful
0619 14:37:27.890241 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":100,"threads_started":1}
I20260619 14:37:27.892647 1787 tablet_service.cc:1467] Processing CreateTablet for tablet 9699687751d6445890255e2e1a78d618 (DEFAULT_TABLE table=test-table [id=997e6f89adcb4f09b78dc6a7abfa0000]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:37:27.892755 1927 tablet_service.cc:1467] Processing CreateTablet for tablet 9699687751d6445890255e2e1a78d618 (DEFAULT_TABLE table=test-table [id=997e6f89adcb4f09b78dc6a7abfa0000]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:37:27.892863 1647 tablet_service.cc:1467] Processing CreateTablet for tablet 9699687751d6445890255e2e1a78d618 (DEFAULT_TABLE table=test-table [id=997e6f89adcb4f09b78dc6a7abfa0000]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:37:27.893698 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9699687751d6445890255e2e1a78d618. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:27.893733 1927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9699687751d6445890255e2e1a78d618. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:27.893728 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9699687751d6445890255e2e1a78d618. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:27.900285 2016 tablet_bootstrap.cc:492] T 9699687751d6445890255e2e1a78d618 P 422fcf83e69b42988c71e1a5a7774760: Bootstrap starting.
I20260619 14:37:27.901345 2018 tablet_bootstrap.cc:492] T 9699687751d6445890255e2e1a78d618 P 6b28eaa67a5d4711a16af5366ff2b67f: Bootstrap starting.
I20260619 14:37:27.902578 2016 tablet_bootstrap.cc:654] T 9699687751d6445890255e2e1a78d618 P 422fcf83e69b42988c71e1a5a7774760: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:27.903327 2018 tablet_bootstrap.cc:654] T 9699687751d6445890255e2e1a78d618 P 6b28eaa67a5d4711a16af5366ff2b67f: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:27.903399 2016 log.cc:824] T 9699687751d6445890255e2e1a78d618 P 422fcf83e69b42988c71e1a5a7774760: Log is configured to *not* fsync() on all Append() calls
I20260619 14:37:27.904057 2017 tablet_bootstrap.cc:492] T 9699687751d6445890255e2e1a78d618 P aee5e9280b8a48b6ada6dd7286f51706: Bootstrap starting.
I20260619 14:37:27.904143 2018 log.cc:824] T 9699687751d6445890255e2e1a78d618 P 6b28eaa67a5d4711a16af5366ff2b67f: Log is configured to *not* fsync() on all Append() calls
I20260619 14:37:27.905441 2016 tablet_bootstrap.cc:492] T 9699687751d6445890255e2e1a78d618 P 422fcf83e69b42988c71e1a5a7774760: No bootstrap required, opened a new log
I20260619 14:37:27.905632 2016 ts_tablet_manager.cc:1397] T 9699687751d6445890255e2e1a78d618 P 422fcf83e69b42988c71e1a5a7774760: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 14:37:27.905984 2017 tablet_bootstrap.cc:654] T 9699687751d6445890255e2e1a78d618 P aee5e9280b8a48b6ada6dd7286f51706: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:27.906260 2018 tablet_bootstrap.cc:492] T 9699687751d6445890255e2e1a78d618 P 6b28eaa67a5d4711a16af5366ff2b67f: No bootstrap required, opened a new log
I20260619 14:37:27.906432 2018 ts_tablet_manager.cc:1397] T 9699687751d6445890255e2e1a78d618 P 6b28eaa67a5d4711a16af5366ff2b67f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:37:27.906658 2017 log.cc:824] T 9699687751d6445890255e2e1a78d618 P aee5e9280b8a48b6ada6dd7286f51706: Log is configured to *not* fsync() on all Append() calls
I20260619 14:37:27.908075 2017 tablet_bootstrap.cc:492] T 9699687751d6445890255e2e1a78d618 P aee5e9280b8a48b6ada6dd7286f51706: No bootstrap required, opened a new log
I20260619 14:37:27.908233 2017 ts_tablet_manager.cc:1397] T 9699687751d6445890255e2e1a78d618 P aee5e9280b8a48b6ada6dd7286f51706: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260619 14:37:27.909420 2016 raft_consensus.cc:348] T 9699687751d6445890255e2e1a78d618 P 422fcf83e69b42988c71e1a5a7774760 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b28eaa67a5d4711a16af5366ff2b67f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37667 } } peers { permanent_uuid: "aee5e9280b8a48b6ada6dd7286f51706" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42729 } } peers { permanent_uuid: "422fcf83e69b42988c71e1a5a7774760" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39197 } }
I20260619 14:37:27.909682 2016 raft_consensus.cc:374] T 9699687751d6445890255e2e1a78d618 P 422fcf83e69b42988c71e1a5a7774760 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:27.909737 2016 raft_consensus.cc:729] T 9699687751d6445890255e2e1a78d618 P 422fcf83e69b42988c71e1a5a7774760 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 422fcf83e69b42988c71e1a5a7774760, State: Initialized, Role: FOLLOWER
I20260619 14:37:27.910012 2018 raft_consensus.cc:348] T 9699687751d6445890255e2e1a78d618 P 6b28eaa67a5d4711a16af5366ff2b67f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b28eaa67a5d4711a16af5366ff2b67f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37667 } } peers { permanent_uuid: "aee5e9280b8a48b6ada6dd7286f51706" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42729 } } peers { permanent_uuid: "422fcf83e69b42988c71e1a5a7774760" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39197 } }
I20260619 14:37:27.910151 2016 consensus_queue.cc:260] T 9699687751d6445890255e2e1a78d618 P 422fcf83e69b42988c71e1a5a7774760 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b28eaa67a5d4711a16af5366ff2b67f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37667 } } peers { permanent_uuid: "aee5e9280b8a48b6ada6dd7286f51706" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42729 } } peers { permanent_uuid: "422fcf83e69b42988c71e1a5a7774760" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39197 } }
I20260619 14:37:27.910236 2018 raft_consensus.cc:374] T 9699687751d6445890255e2e1a78d618 P 6b28eaa67a5d4711a16af5366ff2b67f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:27.910284 2018 raft_consensus.cc:729] T 9699687751d6445890255e2e1a78d618 P 6b28eaa67a5d4711a16af5366ff2b67f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b28eaa67a5d4711a16af5366ff2b67f, State: Initialized, Role: FOLLOWER
I20260619 14:37:27.910701 2018 consensus_queue.cc:260] T 9699687751d6445890255e2e1a78d618 P 6b28eaa67a5d4711a16af5366ff2b67f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b28eaa67a5d4711a16af5366ff2b67f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37667 } } peers { permanent_uuid: "aee5e9280b8a48b6ada6dd7286f51706" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42729 } } peers { permanent_uuid: "422fcf83e69b42988c71e1a5a7774760" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39197 } }
I20260619 14:37:27.911072 2016 ts_tablet_manager.cc:1428] T 9699687751d6445890255e2e1a78d618 P 422fcf83e69b42988c71e1a5a7774760: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 14:37:27.911399 1713 heartbeater.cc:503] Master 127.0.98.62:43579 was elected leader, sending a full tablet report...
I20260619 14:37:27.911409 1853 heartbeater.cc:503] Master 127.0.98.62:43579 was elected leader, sending a full tablet report...
I20260619 14:37:27.911279 2017 raft_consensus.cc:348] T 9699687751d6445890255e2e1a78d618 P aee5e9280b8a48b6ada6dd7286f51706 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b28eaa67a5d4711a16af5366ff2b67f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37667 } } peers { permanent_uuid: "aee5e9280b8a48b6ada6dd7286f51706" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42729 } } peers { permanent_uuid: "422fcf83e69b42988c71e1a5a7774760" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39197 } }
I20260619 14:37:27.911494 2017 raft_consensus.cc:374] T 9699687751d6445890255e2e1a78d618 P aee5e9280b8a48b6ada6dd7286f51706 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:27.911542 2017 raft_consensus.cc:729] T 9699687751d6445890255e2e1a78d618 P aee5e9280b8a48b6ada6dd7286f51706 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aee5e9280b8a48b6ada6dd7286f51706, State: Initialized, Role: FOLLOWER
I20260619 14:37:27.911578 2018 ts_tablet_manager.cc:1428] T 9699687751d6445890255e2e1a78d618 P 6b28eaa67a5d4711a16af5366ff2b67f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 14:37:27.911916 2017 consensus_queue.cc:260] T 9699687751d6445890255e2e1a78d618 P aee5e9280b8a48b6ada6dd7286f51706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b28eaa67a5d4711a16af5366ff2b67f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37667 } } peers { permanent_uuid: "aee5e9280b8a48b6ada6dd7286f51706" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42729 } } peers { permanent_uuid: "422fcf83e69b42988c71e1a5a7774760" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39197 } }
I20260619 14:37:27.912770 2017 ts_tablet_manager.cc:1428] T 9699687751d6445890255e2e1a78d618 P aee5e9280b8a48b6ada6dd7286f51706: Time spent starting tablet: real 0.004s user 0.005s sys 0.001s
I20260619 14:37:27.913045 1993 heartbeater.cc:503] Master 127.0.98.62:43579 was elected leader, sending a full tablet report...
I20260619 14:37:27.917968 2023 raft_consensus.cc:482] T 9699687751d6445890255e2e1a78d618 P 6b28eaa67a5d4711a16af5366ff2b67f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:37:27.918079 2023 raft_consensus.cc:504] T 9699687751d6445890255e2e1a78d618 P 6b28eaa67a5d4711a16af5366ff2b67f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b28eaa67a5d4711a16af5366ff2b67f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37667 } } peers { permanent_uuid: "aee5e9280b8a48b6ada6dd7286f51706" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42729 } } peers { permanent_uuid: "422fcf83e69b42988c71e1a5a7774760" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39197 } }
I20260619 14:37:27.918915 2023 leader_election.cc:288] T 9699687751d6445890255e2e1a78d618 P 6b28eaa67a5d4711a16af5366ff2b67f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aee5e9280b8a48b6ada6dd7286f51706 (127.0.98.3:42729), 422fcf83e69b42988c71e1a5a7774760 (127.0.98.1:39197)
I20260619 14:37:27.922343 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:27.919164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43983 (local address 127.0.98.3:42729)
0619 14:37:27.919287 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:27.919290 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:27.919303 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:37:27.919412 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:27.919415 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:27.919456 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:27.919516 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:37:27.919754 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:27.920279 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:27.921428 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:27.921819 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:27.921909 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:27.921965 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:27.922172 (+ 207us) server_negotiation.cc:299] Negotiation successful
0619 14:37:27.922233 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 14:37:27.922344 1746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:27.919113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42729 (local address 127.0.98.2:43983)
0619 14:37:27.919220 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:27.919235 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:27.919311 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:27.919541 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:27.919544 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:27.919561 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:37:27.919662 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:27.919667 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:27.920371 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:27.920374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:27.921318 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:27.921328 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:27.921922 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:27.921926 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:27.922072 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:27.922089 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 14:37:27.922151 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 14:37:27.922221 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":33}
I20260619 14:37:27.922884 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9699687751d6445890255e2e1a78d618" candidate_uuid: "6b28eaa67a5d4711a16af5366ff2b67f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aee5e9280b8a48b6ada6dd7286f51706" is_pre_election: true
I20260619 14:37:27.923117 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:27.919470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45235 (local address 127.0.98.1:39197)
0619 14:37:27.919632 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:27.919636 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:27.919703 (+ 67us) server_negotiation.cc:407] Connection header received
0619 14:37:27.919831 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:27.919834 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:27.919871 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:27.919928 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:37:27.920158 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:27.920808 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:27.922175 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:27.922642 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:27.922736 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:27.922788 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:27.922962 (+ 174us) server_negotiation.cc:299] Negotiation successful
0619 14:37:27.923021 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260619 14:37:27.923106 1947 raft_consensus.cc:2454] T 9699687751d6445890255e2e1a78d618 P aee5e9280b8a48b6ada6dd7286f51706 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b28eaa67a5d4711a16af5366ff2b67f in term 0.
I20260619 14:37:27.923146 2025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:27.919375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39197 (local address 127.0.98.2:45235)
0619 14:37:27.919637 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:27.919649 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:27.919739 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:27.919944 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:27.919946 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:27.919961 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:37:27.920071 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:27.920076 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:27.920942 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:27.920945 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:27.922069 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:27.922078 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:27.922766 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:27.922770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:27.922862 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:27.922881 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 14:37:27.922943 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 14:37:27.923004 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":21,"thread_start_us":90,"threads_started":1}
I20260619 14:37:27.923503 1741 leader_election.cc:302] T 9699687751d6445890255e2e1a78d618 P 6b28eaa67a5d4711a16af5366ff2b67f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b28eaa67a5d4711a16af5366ff2b67f, aee5e9280b8a48b6ada6dd7286f51706; no voters:
I20260619 14:37:27.923578 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9699687751d6445890255e2e1a78d618" candidate_uuid: "6b28eaa67a5d4711a16af5366ff2b67f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "422fcf83e69b42988c71e1a5a7774760" is_pre_election: true
I20260619 14:37:27.923739 2023 raft_consensus.cc:2781] T 9699687751d6445890255e2e1a78d618 P 6b28eaa67a5d4711a16af5366ff2b67f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:37:27.923815 2023 raft_consensus.cc:482] T 9699687751d6445890255e2e1a78d618 P 6b28eaa67a5d4711a16af5366ff2b67f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:37:27.923803 1667 raft_consensus.cc:2454] T 9699687751d6445890255e2e1a78d618 P 422fcf83e69b42988c71e1a5a7774760 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b28eaa67a5d4711a16af5366ff2b67f in term 0.
I20260619 14:37:27.923863 2023 raft_consensus.cc:3037] T 9699687751d6445890255e2e1a78d618 P 6b28eaa67a5d4711a16af5366ff2b67f [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:27.924850 2023 raft_consensus.cc:504] T 9699687751d6445890255e2e1a78d618 P 6b28eaa67a5d4711a16af5366ff2b67f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b28eaa67a5d4711a16af5366ff2b67f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37667 } } peers { permanent_uuid: "aee5e9280b8a48b6ada6dd7286f51706" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42729 } } peers { permanent_uuid: "422fcf83e69b42988c71e1a5a7774760" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39197 } }
I20260619 14:37:27.925125 2023 leader_election.cc:288] T 9699687751d6445890255e2e1a78d618 P 6b28eaa67a5d4711a16af5366ff2b67f [CANDIDATE]: Term 1 election: Requested vote from peers aee5e9280b8a48b6ada6dd7286f51706 (127.0.98.3:42729), 422fcf83e69b42988c71e1a5a7774760 (127.0.98.1:39197)
I20260619 14:37:27.925390 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9699687751d6445890255e2e1a78d618" candidate_uuid: "6b28eaa67a5d4711a16af5366ff2b67f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aee5e9280b8a48b6ada6dd7286f51706"
I20260619 14:37:27.925427 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9699687751d6445890255e2e1a78d618" candidate_uuid: "6b28eaa67a5d4711a16af5366ff2b67f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "422fcf83e69b42988c71e1a5a7774760"
I20260619 14:37:27.925495 1947 raft_consensus.cc:3037] T 9699687751d6445890255e2e1a78d618 P aee5e9280b8a48b6ada6dd7286f51706 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:27.925509 1667 raft_consensus.cc:3037] T 9699687751d6445890255e2e1a78d618 P 422fcf83e69b42988c71e1a5a7774760 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:27.926647 1667 raft_consensus.cc:2454] T 9699687751d6445890255e2e1a78d618 P 422fcf83e69b42988c71e1a5a7774760 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b28eaa67a5d4711a16af5366ff2b67f in term 1.
I20260619 14:37:27.926647 1947 raft_consensus.cc:2454] T 9699687751d6445890255e2e1a78d618 P aee5e9280b8a48b6ada6dd7286f51706 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b28eaa67a5d4711a16af5366ff2b67f in term 1.
I20260619 14:37:27.926901 1741 leader_election.cc:302] T 9699687751d6445890255e2e1a78d618 P 6b28eaa67a5d4711a16af5366ff2b67f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 422fcf83e69b42988c71e1a5a7774760, 6b28eaa67a5d4711a16af5366ff2b67f; no voters:
I20260619 14:37:27.927052 2023 raft_consensus.cc:2781] T 9699687751d6445890255e2e1a78d618 P 6b28eaa67a5d4711a16af5366ff2b67f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:37:27.927220 2023 raft_consensus.cc:686] T 9699687751d6445890255e2e1a78d618 P 6b28eaa67a5d4711a16af5366ff2b67f [term 1 LEADER]: Becoming Leader. State: Replica: 6b28eaa67a5d4711a16af5366ff2b67f, State: Running, Role: LEADER
I20260619 14:37:27.927488 2023 consensus_queue.cc:237] T 9699687751d6445890255e2e1a78d618 P 6b28eaa67a5d4711a16af5366ff2b67f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b28eaa67a5d4711a16af5366ff2b67f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37667 } } peers { permanent_uuid: "aee5e9280b8a48b6ada6dd7286f51706" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42729 } } peers { permanent_uuid: "422fcf83e69b42988c71e1a5a7774760" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39197 } }
I20260619 14:37:27.930217 1509 catalog_manager.cc:5495] T 9699687751d6445890255e2e1a78d618 P 6b28eaa67a5d4711a16af5366ff2b67f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b28eaa67a5d4711a16af5366ff2b67f (127.0.98.2). New cstate: current_term: 1 leader_uuid: "6b28eaa67a5d4711a16af5366ff2b67f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b28eaa67a5d4711a16af5366ff2b67f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aee5e9280b8a48b6ada6dd7286f51706" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "422fcf83e69b42988c71e1a5a7774760" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39197 } health_report { overall_health: UNKNOWN } } }
W20260619 14:37:27.943264 1714 tablet.cc:2369] T 9699687751d6445890255e2e1a78d618 P 422fcf83e69b42988c71e1a5a7774760: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:37:27.955107 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:27.951601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48724 (local address 127.0.98.2:37667)
0619 14:37:27.951718 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:27.951722 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:27.951736 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:37:27.951931 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:27.951934 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:27.951987 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:27.952068 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:37:27.952324 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:27.952865 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:27.953720 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:27.953946 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:27.954032 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:27.954346 (+ 314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:37:27.954734 (+ 388us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:37:27.954841 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:27.954922 (+ 81us) server_negotiation.cc:299] Negotiation successful
0619 14:37:27.954978 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 14:37:27.960543 1667 raft_consensus.cc:1261] T 9699687751d6445890255e2e1a78d618 P 422fcf83e69b42988c71e1a5a7774760 [term 1 FOLLOWER]: Refusing update from remote peer 6b28eaa67a5d4711a16af5366ff2b67f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:37:27.960822 1947 raft_consensus.cc:1261] T 9699687751d6445890255e2e1a78d618 P aee5e9280b8a48b6ada6dd7286f51706 [term 1 FOLLOWER]: Refusing update from remote peer 6b28eaa67a5d4711a16af5366ff2b67f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:37:27.961308 2026 consensus_queue.cc:1035] T 9699687751d6445890255e2e1a78d618 P 6b28eaa67a5d4711a16af5366ff2b67f [LEADER]: Connected to new peer: Peer: permanent_uuid: "422fcf83e69b42988c71e1a5a7774760" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:37:27.961465 2023 consensus_queue.cc:1035] T 9699687751d6445890255e2e1a78d618 P 6b28eaa67a5d4711a16af5366ff2b67f [LEADER]: Connected to new peer: Peer: permanent_uuid: "aee5e9280b8a48b6ada6dd7286f51706" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:37:27.972154 2031 mvcc.cc:205] Tried to move back new op lower bound from 7298579857239486464 to 7298579857113235456. Current Snapshot: MvccSnapshot[applied={T|T < 7298579857239486464 or (T in {7298579857239486464})}]
I20260619 14:37:27.979056 1509 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56294:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:37:27.979194 1509 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56294:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:37:27.981357 1509 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 82fc67ef30144ceabdac56aab5b87dc8: Sending DeleteTablet for 3 replicas of tablet 9699687751d6445890255e2e1a78d618
I20260619 14:37:27.982015 1787 tablet_service.cc:1514] Processing DeleteTablet for tablet 9699687751d6445890255e2e1a78d618 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:37:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48714
I20260619 14:37:27.982060 1647 tablet_service.cc:1514] Processing DeleteTablet for tablet 9699687751d6445890255e2e1a78d618 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:37:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56214
I20260619 14:37:27.982079 1927 tablet_service.cc:1514] Processing DeleteTablet for tablet 9699687751d6445890255e2e1a78d618 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:37:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59774
I20260619 14:37:27.982517 2038 tablet_replica.cc:330] T 9699687751d6445890255e2e1a78d618 P 6b28eaa67a5d4711a16af5366ff2b67f: stopping tablet replica
I20260619 14:37:27.982640 2039 tablet_replica.cc:330] T 9699687751d6445890255e2e1a78d618 P aee5e9280b8a48b6ada6dd7286f51706: stopping tablet replica
I20260619 14:37:27.982733 2040 tablet_replica.cc:330] T 9699687751d6445890255e2e1a78d618 P 422fcf83e69b42988c71e1a5a7774760: stopping tablet replica
I20260619 14:37:27.983012 2040 raft_consensus.cc:2229] T 9699687751d6445890255e2e1a78d618 P 422fcf83e69b42988c71e1a5a7774760 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:37:27.983111 2039 raft_consensus.cc:2229] T 9699687751d6445890255e2e1a78d618 P aee5e9280b8a48b6ada6dd7286f51706 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:37:27.983285 1509 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56294:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 14:37:27.983422 2039 raft_consensus.cc:2258] T 9699687751d6445890255e2e1a78d618 P aee5e9280b8a48b6ada6dd7286f51706 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 14:37:27.983662 1509 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:37:27.984498 2038 raft_consensus.cc:2229] T 9699687751d6445890255e2e1a78d618 P 6b28eaa67a5d4711a16af5366ff2b67f [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:37:27.984866 2040 raft_consensus.cc:2258] T 9699687751d6445890255e2e1a78d618 P 422fcf83e69b42988c71e1a5a7774760 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:37:27.985277 2039 ts_tablet_manager.cc:1905] T 9699687751d6445890255e2e1a78d618 P aee5e9280b8a48b6ada6dd7286f51706: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:37:27.985328 2038 raft_consensus.cc:2258] T 9699687751d6445890255e2e1a78d618 P 6b28eaa67a5d4711a16af5366ff2b67f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:37:27.986999 2040 ts_tablet_manager.cc:1905] T 9699687751d6445890255e2e1a78d618 P 422fcf83e69b42988c71e1a5a7774760: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:37:27.988397 1787 tablet_service.cc:1467] Processing CreateTablet for tablet aa540acd584046dea8c034702e8631f5 (DEFAULT_TABLE table=test-table [id=eb6ac2fc22134316b3d883279d8cdbc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:37:27.988510 1927 tablet_service.cc:1467] Processing CreateTablet for tablet aa540acd584046dea8c034702e8631f5 (DEFAULT_TABLE table=test-table [id=eb6ac2fc22134316b3d883279d8cdbc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:37:27.988651 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa540acd584046dea8c034702e8631f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:27.988804 2038 ts_tablet_manager.cc:1905] T 9699687751d6445890255e2e1a78d618 P 6b28eaa67a5d4711a16af5366ff2b67f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:37:27.989068 1927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa540acd584046dea8c034702e8631f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:27.988296 1647 tablet_service.cc:1467] Processing CreateTablet for tablet aa540acd584046dea8c034702e8631f5 (DEFAULT_TABLE table=test-table [id=eb6ac2fc22134316b3d883279d8cdbc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:37:27.991108 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa540acd584046dea8c034702e8631f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:27.991122 2018 tablet_bootstrap.cc:492] T aa540acd584046dea8c034702e8631f5 P 6b28eaa67a5d4711a16af5366ff2b67f: Bootstrap starting.
I20260619 14:37:27.992038 2018 tablet_bootstrap.cc:654] T aa540acd584046dea8c034702e8631f5 P 6b28eaa67a5d4711a16af5366ff2b67f: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:27.993129 2039 ts_tablet_manager.cc:1918] T 9699687751d6445890255e2e1a78d618 P aee5e9280b8a48b6ada6dd7286f51706: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:37:27.993208 2039 log.cc:1196] T 9699687751d6445890255e2e1a78d618 P aee5e9280b8a48b6ada6dd7286f51706: Deleting WAL directory at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/ts-2/wal/wals/9699687751d6445890255e2e1a78d618
I20260619 14:37:27.993358 2016 tablet_bootstrap.cc:492] T aa540acd584046dea8c034702e8631f5 P 422fcf83e69b42988c71e1a5a7774760: Bootstrap starting.
I20260619 14:37:27.993443 2017 tablet_bootstrap.cc:492] T aa540acd584046dea8c034702e8631f5 P aee5e9280b8a48b6ada6dd7286f51706: Bootstrap starting.
I20260619 14:37:27.993490 2039 ts_tablet_manager.cc:1939] T 9699687751d6445890255e2e1a78d618 P aee5e9280b8a48b6ada6dd7286f51706: Deleting consensus metadata
I20260619 14:37:27.994068 2018 tablet_bootstrap.cc:492] T aa540acd584046dea8c034702e8631f5 P 6b28eaa67a5d4711a16af5366ff2b67f: No bootstrap required, opened a new log
I20260619 14:37:27.994139 2018 ts_tablet_manager.cc:1397] T aa540acd584046dea8c034702e8631f5 P 6b28eaa67a5d4711a16af5366ff2b67f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 14:37:27.994201 2016 tablet_bootstrap.cc:654] T aa540acd584046dea8c034702e8631f5 P 422fcf83e69b42988c71e1a5a7774760: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:27.994563 2018 raft_consensus.cc:348] T aa540acd584046dea8c034702e8631f5 P 6b28eaa67a5d4711a16af5366ff2b67f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422fcf83e69b42988c71e1a5a7774760" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39197 } } peers { permanent_uuid: "6b28eaa67a5d4711a16af5366ff2b67f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37667 } } peers { permanent_uuid: "aee5e9280b8a48b6ada6dd7286f51706" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42729 } }
I20260619 14:37:27.994660 2018 raft_consensus.cc:374] T aa540acd584046dea8c034702e8631f5 P 6b28eaa67a5d4711a16af5366ff2b67f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:27.994689 2018 raft_consensus.cc:729] T aa540acd584046dea8c034702e8631f5 P 6b28eaa67a5d4711a16af5366ff2b67f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b28eaa67a5d4711a16af5366ff2b67f, State: Initialized, Role: FOLLOWER
I20260619 14:37:27.994848 2017 tablet_bootstrap.cc:654] T aa540acd584046dea8c034702e8631f5 P aee5e9280b8a48b6ada6dd7286f51706: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:27.994808 2018 consensus_queue.cc:260] T aa540acd584046dea8c034702e8631f5 P 6b28eaa67a5d4711a16af5366ff2b67f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422fcf83e69b42988c71e1a5a7774760" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39197 } } peers { permanent_uuid: "6b28eaa67a5d4711a16af5366ff2b67f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37667 } } peers { permanent_uuid: "aee5e9280b8a48b6ada6dd7286f51706" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42729 } }
I20260619 14:37:27.995038 2018 ts_tablet_manager.cc:1428] T aa540acd584046dea8c034702e8631f5 P 6b28eaa67a5d4711a16af5366ff2b67f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:37:27.995730 1497 catalog_manager.cc:4841] TS aee5e9280b8a48b6ada6dd7286f51706 (127.0.98.3:42729): tablet 9699687751d6445890255e2e1a78d618 (table test-table [id=997e6f89adcb4f09b78dc6a7abfa0000]) successfully deleted
I20260619 14:37:27.997468 2038 ts_tablet_manager.cc:1918] T 9699687751d6445890255e2e1a78d618 P 6b28eaa67a5d4711a16af5366ff2b67f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:37:27.997622 2038 log.cc:1196] T 9699687751d6445890255e2e1a78d618 P 6b28eaa67a5d4711a16af5366ff2b67f: Deleting WAL directory at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/ts-1/wal/wals/9699687751d6445890255e2e1a78d618
I20260619 14:37:27.998076 2038 ts_tablet_manager.cc:1939] T 9699687751d6445890255e2e1a78d618 P 6b28eaa67a5d4711a16af5366ff2b67f: Deleting consensus metadata
I20260619 14:37:27.998540 2040 ts_tablet_manager.cc:1918] T 9699687751d6445890255e2e1a78d618 P 422fcf83e69b42988c71e1a5a7774760: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:37:27.998643 2040 log.cc:1196] T 9699687751d6445890255e2e1a78d618 P 422fcf83e69b42988c71e1a5a7774760: Deleting WAL directory at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/ts-0/wal/wals/9699687751d6445890255e2e1a78d618
I20260619 14:37:27.998945 2040 ts_tablet_manager.cc:1939] T 9699687751d6445890255e2e1a78d618 P 422fcf83e69b42988c71e1a5a7774760: Deleting consensus metadata
I20260619 14:37:27.999217 1495 catalog_manager.cc:4841] TS 6b28eaa67a5d4711a16af5366ff2b67f (127.0.98.2:37667): tablet 9699687751d6445890255e2e1a78d618 (table test-table [id=997e6f89adcb4f09b78dc6a7abfa0000]) successfully deleted
I20260619 14:37:27.999660 2017 tablet_bootstrap.cc:492] T aa540acd584046dea8c034702e8631f5 P aee5e9280b8a48b6ada6dd7286f51706: No bootstrap required, opened a new log
I20260619 14:37:27.999729 2017 ts_tablet_manager.cc:1397] T aa540acd584046dea8c034702e8631f5 P aee5e9280b8a48b6ada6dd7286f51706: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260619 14:37:28.000078 2017 raft_consensus.cc:348] T aa540acd584046dea8c034702e8631f5 P aee5e9280b8a48b6ada6dd7286f51706 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422fcf83e69b42988c71e1a5a7774760" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39197 } } peers { permanent_uuid: "6b28eaa67a5d4711a16af5366ff2b67f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37667 } } peers { permanent_uuid: "aee5e9280b8a48b6ada6dd7286f51706" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42729 } }
I20260619 14:37:28.000164 2017 raft_consensus.cc:374] T aa540acd584046dea8c034702e8631f5 P aee5e9280b8a48b6ada6dd7286f51706 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:28.000185 2017 raft_consensus.cc:729] T aa540acd584046dea8c034702e8631f5 P aee5e9280b8a48b6ada6dd7286f51706 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aee5e9280b8a48b6ada6dd7286f51706, State: Initialized, Role: FOLLOWER
I20260619 14:37:28.000306 2017 consensus_queue.cc:260] T aa540acd584046dea8c034702e8631f5 P aee5e9280b8a48b6ada6dd7286f51706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422fcf83e69b42988c71e1a5a7774760" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39197 } } peers { permanent_uuid: "6b28eaa67a5d4711a16af5366ff2b67f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37667 } } peers { permanent_uuid: "aee5e9280b8a48b6ada6dd7286f51706" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42729 } }
I20260619 14:37:28.000521 2017 ts_tablet_manager.cc:1428] T aa540acd584046dea8c034702e8631f5 P aee5e9280b8a48b6ada6dd7286f51706: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 14:37:28.001348 1497 catalog_manager.cc:4841] TS 422fcf83e69b42988c71e1a5a7774760 (127.0.98.1:39197): tablet 9699687751d6445890255e2e1a78d618 (table test-table [id=997e6f89adcb4f09b78dc6a7abfa0000]) successfully deleted
I20260619 14:37:28.005683 2016 tablet_bootstrap.cc:492] T aa540acd584046dea8c034702e8631f5 P 422fcf83e69b42988c71e1a5a7774760: No bootstrap required, opened a new log
I20260619 14:37:28.005779 2016 ts_tablet_manager.cc:1397] T aa540acd584046dea8c034702e8631f5 P 422fcf83e69b42988c71e1a5a7774760: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s
I20260619 14:37:28.006203 2016 raft_consensus.cc:348] T aa540acd584046dea8c034702e8631f5 P 422fcf83e69b42988c71e1a5a7774760 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422fcf83e69b42988c71e1a5a7774760" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39197 } } peers { permanent_uuid: "6b28eaa67a5d4711a16af5366ff2b67f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37667 } } peers { permanent_uuid: "aee5e9280b8a48b6ada6dd7286f51706" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42729 } }
I20260619 14:37:28.006326 2016 raft_consensus.cc:374] T aa540acd584046dea8c034702e8631f5 P 422fcf83e69b42988c71e1a5a7774760 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:28.006364 2016 raft_consensus.cc:729] T aa540acd584046dea8c034702e8631f5 P 422fcf83e69b42988c71e1a5a7774760 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 422fcf83e69b42988c71e1a5a7774760, State: Initialized, Role: FOLLOWER
I20260619 14:37:28.006475 2016 consensus_queue.cc:260] T aa540acd584046dea8c034702e8631f5 P 422fcf83e69b42988c71e1a5a7774760 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422fcf83e69b42988c71e1a5a7774760" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39197 } } peers { permanent_uuid: "6b28eaa67a5d4711a16af5366ff2b67f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37667 } } peers { permanent_uuid: "aee5e9280b8a48b6ada6dd7286f51706" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42729 } }
I20260619 14:37:28.006722 2016 ts_tablet_manager.cc:1428] T aa540acd584046dea8c034702e8631f5 P 422fcf83e69b42988c71e1a5a7774760: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260619 14:37:28.090274 1994 tablet.cc:2369] T aa540acd584046dea8c034702e8631f5 P aee5e9280b8a48b6ada6dd7286f51706: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:37:28.137346 1854 tablet.cc:2369] T aa540acd584046dea8c034702e8631f5 P 6b28eaa67a5d4711a16af5366ff2b67f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:37:28.271067 2024 raft_consensus.cc:482] T aa540acd584046dea8c034702e8631f5 P aee5e9280b8a48b6ada6dd7286f51706 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:37:28.271225 2024 raft_consensus.cc:504] T aa540acd584046dea8c034702e8631f5 P aee5e9280b8a48b6ada6dd7286f51706 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422fcf83e69b42988c71e1a5a7774760" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39197 } } peers { permanent_uuid: "6b28eaa67a5d4711a16af5366ff2b67f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37667 } } peers { permanent_uuid: "aee5e9280b8a48b6ada6dd7286f51706" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42729 } }
I20260619 14:37:28.272243 2024 leader_election.cc:288] T aa540acd584046dea8c034702e8631f5 P aee5e9280b8a48b6ada6dd7286f51706 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 422fcf83e69b42988c71e1a5a7774760 (127.0.98.1:39197), 6b28eaa67a5d4711a16af5366ff2b67f (127.0.98.2:37667)
I20260619 14:37:28.275736 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:28.272578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33795 (local address 127.0.98.2:37667)
0619 14:37:28.272692 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:28.272695 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:28.272812 (+ 117us) server_negotiation.cc:407] Connection header received
0619 14:37:28.272995 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:28.272998 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:28.273039 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:28.273099 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:37:28.273355 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:28.273842 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:28.274847 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:28.275244 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:28.275343 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:28.275407 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:28.275561 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 14:37:28.275628 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 14:37:28.275763 1886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:28.272495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37667 (local address 127.0.98.3:33795)
0619 14:37:28.272703 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:28.272742 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:28.272860 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:28.273126 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:28.273129 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:28.273145 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:37:28.273251 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:28.273257 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:28.273937 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:28.273940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:28.274719 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:28.274727 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:28.275354 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:28.275357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:28.275451 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:28.275472 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 14:37:28.275537 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 14:37:28.275606 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139}
I20260619 14:37:28.275856 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:28.272832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46385 (local address 127.0.98.1:39197)
0619 14:37:28.272949 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:28.272953 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:28.272991 (+ 38us) server_negotiation.cc:407] Connection header received
0619 14:37:28.273160 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:28.273162 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:28.273202 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:28.273263 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:37:28.273514 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:28.274018 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:28.274988 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:28.275361 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:28.275446 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:28.275502 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:28.275683 (+ 181us) server_negotiation.cc:299] Negotiation successful
0619 14:37:28.275746 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 14:37:28.275921 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:28.272495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39197 (local address 127.0.98.3:46385)
0619 14:37:28.272904 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:28.272918 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:28.273039 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:28.273279 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:28.273281 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:28.273297 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:37:28.273420 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:28.273425 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:28.274112 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:28.274115 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:28.274879 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:28.274886 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:28.275476 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:28.275480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:28.275570 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:28.275599 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 14:37:28.275661 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 14:37:28.275722 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"mutex_wait_us":28,"thread_start_us":68,"threads_started":1}
I20260619 14:37:28.276293 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa540acd584046dea8c034702e8631f5" candidate_uuid: "aee5e9280b8a48b6ada6dd7286f51706" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "422fcf83e69b42988c71e1a5a7774760" is_pre_election: true
I20260619 14:37:28.276419 1667 raft_consensus.cc:2454] T aa540acd584046dea8c034702e8631f5 P 422fcf83e69b42988c71e1a5a7774760 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aee5e9280b8a48b6ada6dd7286f51706 in term 0.
I20260619 14:37:28.276453 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa540acd584046dea8c034702e8631f5" candidate_uuid: "aee5e9280b8a48b6ada6dd7286f51706" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b28eaa67a5d4711a16af5366ff2b67f" is_pre_election: true
I20260619 14:37:28.276690 1807 raft_consensus.cc:2454] T aa540acd584046dea8c034702e8631f5 P 6b28eaa67a5d4711a16af5366ff2b67f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aee5e9280b8a48b6ada6dd7286f51706 in term 0.
I20260619 14:37:28.276777 1881 leader_election.cc:302] T aa540acd584046dea8c034702e8631f5 P aee5e9280b8a48b6ada6dd7286f51706 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 422fcf83e69b42988c71e1a5a7774760, aee5e9280b8a48b6ada6dd7286f51706; no voters:
I20260619 14:37:28.277272 2024 raft_consensus.cc:2781] T aa540acd584046dea8c034702e8631f5 P aee5e9280b8a48b6ada6dd7286f51706 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:37:28.277323 2024 raft_consensus.cc:482] T aa540acd584046dea8c034702e8631f5 P aee5e9280b8a48b6ada6dd7286f51706 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:37:28.277359 2024 raft_consensus.cc:3037] T aa540acd584046dea8c034702e8631f5 P aee5e9280b8a48b6ada6dd7286f51706 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:28.278368 2024 raft_consensus.cc:504] T aa540acd584046dea8c034702e8631f5 P aee5e9280b8a48b6ada6dd7286f51706 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422fcf83e69b42988c71e1a5a7774760" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39197 } } peers { permanent_uuid: "6b28eaa67a5d4711a16af5366ff2b67f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37667 } } peers { permanent_uuid: "aee5e9280b8a48b6ada6dd7286f51706" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42729 } }
I20260619 14:37:28.278671 2024 leader_election.cc:288] T aa540acd584046dea8c034702e8631f5 P aee5e9280b8a48b6ada6dd7286f51706 [CANDIDATE]: Term 1 election: Requested vote from peers 422fcf83e69b42988c71e1a5a7774760 (127.0.98.1:39197), 6b28eaa67a5d4711a16af5366ff2b67f (127.0.98.2:37667)
I20260619 14:37:28.278941 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa540acd584046dea8c034702e8631f5" candidate_uuid: "aee5e9280b8a48b6ada6dd7286f51706" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "422fcf83e69b42988c71e1a5a7774760"
I20260619 14:37:28.278980 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa540acd584046dea8c034702e8631f5" candidate_uuid: "aee5e9280b8a48b6ada6dd7286f51706" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b28eaa67a5d4711a16af5366ff2b67f"
I20260619 14:37:28.279062 1807 raft_consensus.cc:3037] T aa540acd584046dea8c034702e8631f5 P 6b28eaa67a5d4711a16af5366ff2b67f [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:28.279062 1667 raft_consensus.cc:3037] T aa540acd584046dea8c034702e8631f5 P 422fcf83e69b42988c71e1a5a7774760 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:28.280092 1667 raft_consensus.cc:2454] T aa540acd584046dea8c034702e8631f5 P 422fcf83e69b42988c71e1a5a7774760 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aee5e9280b8a48b6ada6dd7286f51706 in term 1.
I20260619 14:37:28.280092 1807 raft_consensus.cc:2454] T aa540acd584046dea8c034702e8631f5 P 6b28eaa67a5d4711a16af5366ff2b67f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aee5e9280b8a48b6ada6dd7286f51706 in term 1.
I20260619 14:37:28.280368 1879 leader_election.cc:302] T aa540acd584046dea8c034702e8631f5 P aee5e9280b8a48b6ada6dd7286f51706 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 422fcf83e69b42988c71e1a5a7774760, 6b28eaa67a5d4711a16af5366ff2b67f, aee5e9280b8a48b6ada6dd7286f51706; no voters:
I20260619 14:37:28.280531 2024 raft_consensus.cc:2781] T aa540acd584046dea8c034702e8631f5 P aee5e9280b8a48b6ada6dd7286f51706 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:37:28.280738 2024 raft_consensus.cc:686] T aa540acd584046dea8c034702e8631f5 P aee5e9280b8a48b6ada6dd7286f51706 [term 1 LEADER]: Becoming Leader. State: Replica: aee5e9280b8a48b6ada6dd7286f51706, State: Running, Role: LEADER
I20260619 14:37:28.280997 2024 consensus_queue.cc:237] T aa540acd584046dea8c034702e8631f5 P aee5e9280b8a48b6ada6dd7286f51706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422fcf83e69b42988c71e1a5a7774760" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39197 } } peers { permanent_uuid: "6b28eaa67a5d4711a16af5366ff2b67f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37667 } } peers { permanent_uuid: "aee5e9280b8a48b6ada6dd7286f51706" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42729 } }
I20260619 14:37:28.282928 1509 catalog_manager.cc:5495] T aa540acd584046dea8c034702e8631f5 P aee5e9280b8a48b6ada6dd7286f51706 reported cstate change: term changed from 0 to 1, leader changed from <none> to aee5e9280b8a48b6ada6dd7286f51706 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "aee5e9280b8a48b6ada6dd7286f51706" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422fcf83e69b42988c71e1a5a7774760" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b28eaa67a5d4711a16af5366ff2b67f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aee5e9280b8a48b6ada6dd7286f51706" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42729 } health_report { overall_health: HEALTHY } } }
Jun 19 14:37:28 dist-test-slave-32hj krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879847, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:37:28.383931 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:28.366331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43579 (local address 127.0.0.1:56302)
0619 14:37:28.366591 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:28.366602 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:28.366713 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:28.366949 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:28.366952 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:28.367377 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:28.367466 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:28.367472 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:28.368176 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:28.368179 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:28.368746 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:28.368757 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:28.369147 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:28.369150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:28.369223 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:28.379527 (+ 10304us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:28.379541 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:28.381022 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:28.382461 (+ 1439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:28.382472 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:28.382478 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:28.382702 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:28.382936 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:28.382939 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:28.382941 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:28.383012 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:28.383362 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:28.383370 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:28.383477 (+ 107us) client_negotiation.cc:769] Sending connection context
0619 14:37:28.383571 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 14:37:28.383731 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":97,"threads_started":1}
I20260619 14:37:28.383931 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:28.366367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56302 (local address 127.0.98.62:43579)
0619 14:37:28.366492 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:28.366497 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:28.366671 (+ 174us) server_negotiation.cc:407] Connection header received
0619 14:37:28.366820 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:28.366823 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:28.366862 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:28.366921 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:28.367574 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:28.368083 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:28.368841 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:28.369059 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:28.369139 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:28.381167 (+ 12028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:28.381170 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:28.381172 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:28.381193 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:28.382363 (+ 1170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:28.382788 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:28.382791 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:28.382793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:28.382861 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:28.383085 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:28.383088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:28.383090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:28.383268 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:28.383350 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:28.383639 (+ 289us) server_negotiation.cc:299] Negotiation successful
0619 14:37:28.383784 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":44}
I20260619 14:37:28.387723 1509 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56302:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 14:37:28.388438 1509 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:37:28.392864 1647 tablet_service.cc:1467] Processing CreateTablet for tablet 668e4a1a14d6463ea0f4488477fdb5e5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=758ebe7159894a1eb1d534e4d489885e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 14:37:28.392889 1787 tablet_service.cc:1467] Processing CreateTablet for tablet 668e4a1a14d6463ea0f4488477fdb5e5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=758ebe7159894a1eb1d534e4d489885e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 14:37:28.392987 1927 tablet_service.cc:1467] Processing CreateTablet for tablet 668e4a1a14d6463ea0f4488477fdb5e5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=758ebe7159894a1eb1d534e4d489885e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 14:37:28.393244 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 668e4a1a14d6463ea0f4488477fdb5e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:28.393265 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 668e4a1a14d6463ea0f4488477fdb5e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:28.393303 1927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 668e4a1a14d6463ea0f4488477fdb5e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:28.395790 2018 tablet_bootstrap.cc:492] T 668e4a1a14d6463ea0f4488477fdb5e5 P 6b28eaa67a5d4711a16af5366ff2b67f: Bootstrap starting.
I20260619 14:37:28.395852 2016 tablet_bootstrap.cc:492] T 668e4a1a14d6463ea0f4488477fdb5e5 P 422fcf83e69b42988c71e1a5a7774760: Bootstrap starting.
I20260619 14:37:28.396481 2018 tablet_bootstrap.cc:654] T 668e4a1a14d6463ea0f4488477fdb5e5 P 6b28eaa67a5d4711a16af5366ff2b67f: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:28.396605 2016 tablet_bootstrap.cc:654] T 668e4a1a14d6463ea0f4488477fdb5e5 P 422fcf83e69b42988c71e1a5a7774760: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:28.397116 2017 tablet_bootstrap.cc:492] T 668e4a1a14d6463ea0f4488477fdb5e5 P aee5e9280b8a48b6ada6dd7286f51706: Bootstrap starting.
I20260619 14:37:28.397811 2018 tablet_bootstrap.cc:492] T 668e4a1a14d6463ea0f4488477fdb5e5 P 6b28eaa67a5d4711a16af5366ff2b67f: No bootstrap required, opened a new log
I20260619 14:37:28.397898 2018 ts_tablet_manager.cc:1397] T 668e4a1a14d6463ea0f4488477fdb5e5 P 6b28eaa67a5d4711a16af5366ff2b67f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 14:37:28.397920 2016 tablet_bootstrap.cc:492] T 668e4a1a14d6463ea0f4488477fdb5e5 P 422fcf83e69b42988c71e1a5a7774760: No bootstrap required, opened a new log
I20260619 14:37:28.397939 2017 tablet_bootstrap.cc:654] T 668e4a1a14d6463ea0f4488477fdb5e5 P aee5e9280b8a48b6ada6dd7286f51706: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:28.397993 2016 ts_tablet_manager.cc:1397] T 668e4a1a14d6463ea0f4488477fdb5e5 P 422fcf83e69b42988c71e1a5a7774760: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 14:37:28.398298 2018 raft_consensus.cc:348] T 668e4a1a14d6463ea0f4488477fdb5e5 P 6b28eaa67a5d4711a16af5366ff2b67f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422fcf83e69b42988c71e1a5a7774760" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39197 } } peers { permanent_uuid: "6b28eaa67a5d4711a16af5366ff2b67f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37667 } } peers { permanent_uuid: "aee5e9280b8a48b6ada6dd7286f51706" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42729 } }
I20260619 14:37:28.398358 2016 raft_consensus.cc:348] T 668e4a1a14d6463ea0f4488477fdb5e5 P 422fcf83e69b42988c71e1a5a7774760 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422fcf83e69b42988c71e1a5a7774760" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39197 } } peers { permanent_uuid: "6b28eaa67a5d4711a16af5366ff2b67f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37667 } } peers { permanent_uuid: "aee5e9280b8a48b6ada6dd7286f51706" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42729 } }
I20260619 14:37:28.398435 2018 raft_consensus.cc:374] T 668e4a1a14d6463ea0f4488477fdb5e5 P 6b28eaa67a5d4711a16af5366ff2b67f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:28.398459 2016 raft_consensus.cc:374] T 668e4a1a14d6463ea0f4488477fdb5e5 P 422fcf83e69b42988c71e1a5a7774760 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:28.398473 2018 raft_consensus.cc:729] T 668e4a1a14d6463ea0f4488477fdb5e5 P 6b28eaa67a5d4711a16af5366ff2b67f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b28eaa67a5d4711a16af5366ff2b67f, State: Initialized, Role: FOLLOWER
I20260619 14:37:28.398489 2016 raft_consensus.cc:729] T 668e4a1a14d6463ea0f4488477fdb5e5 P 422fcf83e69b42988c71e1a5a7774760 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 422fcf83e69b42988c71e1a5a7774760, State: Initialized, Role: FOLLOWER
I20260619 14:37:28.398587 2018 consensus_queue.cc:260] T 668e4a1a14d6463ea0f4488477fdb5e5 P 6b28eaa67a5d4711a16af5366ff2b67f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422fcf83e69b42988c71e1a5a7774760" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39197 } } peers { permanent_uuid: "6b28eaa67a5d4711a16af5366ff2b67f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37667 } } peers { permanent_uuid: "aee5e9280b8a48b6ada6dd7286f51706" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42729 } }
I20260619 14:37:28.398609 2016 consensus_queue.cc:260] T 668e4a1a14d6463ea0f4488477fdb5e5 P 422fcf83e69b42988c71e1a5a7774760 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422fcf83e69b42988c71e1a5a7774760" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39197 } } peers { permanent_uuid: "6b28eaa67a5d4711a16af5366ff2b67f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37667 } } peers { permanent_uuid: "aee5e9280b8a48b6ada6dd7286f51706" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42729 } }
I20260619 14:37:28.398841 2016 ts_tablet_manager.cc:1428] T 668e4a1a14d6463ea0f4488477fdb5e5 P 422fcf83e69b42988c71e1a5a7774760: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:37:28.398852 2018 ts_tablet_manager.cc:1428] T 668e4a1a14d6463ea0f4488477fdb5e5 P 6b28eaa67a5d4711a16af5366ff2b67f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 14:37:28.399075 2017 tablet_bootstrap.cc:492] T 668e4a1a14d6463ea0f4488477fdb5e5 P aee5e9280b8a48b6ada6dd7286f51706: No bootstrap required, opened a new log
I20260619 14:37:28.399019 2022 tablet_replica.cc:439] T 668e4a1a14d6463ea0f4488477fdb5e5 P 422fcf83e69b42988c71e1a5a7774760: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422fcf83e69b42988c71e1a5a7774760" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39197 } } peers { permanent_uuid: "6b28eaa67a5d4711a16af5366ff2b67f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37667 } } peers { permanent_uuid: "aee5e9280b8a48b6ada6dd7286f51706" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42729 } } }
I20260619 14:37:28.399052 2026 tablet_replica.cc:439] T 668e4a1a14d6463ea0f4488477fdb5e5 P 6b28eaa67a5d4711a16af5366ff2b67f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422fcf83e69b42988c71e1a5a7774760" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39197 } } peers { permanent_uuid: "6b28eaa67a5d4711a16af5366ff2b67f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37667 } } peers { permanent_uuid: "aee5e9280b8a48b6ada6dd7286f51706" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42729 } } }
I20260619 14:37:28.399135 2022 tablet_replica.cc:442] T 668e4a1a14d6463ea0f4488477fdb5e5 P 422fcf83e69b42988c71e1a5a7774760: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:37:28.399140 2017 ts_tablet_manager.cc:1397] T 668e4a1a14d6463ea0f4488477fdb5e5 P aee5e9280b8a48b6ada6dd7286f51706: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260619 14:37:28.399158 2026 tablet_replica.cc:442] T 668e4a1a14d6463ea0f4488477fdb5e5 P 6b28eaa67a5d4711a16af5366ff2b67f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:37:28.399533 2017 raft_consensus.cc:348] T 668e4a1a14d6463ea0f4488477fdb5e5 P aee5e9280b8a48b6ada6dd7286f51706 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422fcf83e69b42988c71e1a5a7774760" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39197 } } peers { permanent_uuid: "6b28eaa67a5d4711a16af5366ff2b67f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37667 } } peers { permanent_uuid: "aee5e9280b8a48b6ada6dd7286f51706" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42729 } }
I20260619 14:37:28.399628 2017 raft_consensus.cc:374] T 668e4a1a14d6463ea0f4488477fdb5e5 P aee5e9280b8a48b6ada6dd7286f51706 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:28.399659 2017 raft_consensus.cc:729] T 668e4a1a14d6463ea0f4488477fdb5e5 P aee5e9280b8a48b6ada6dd7286f51706 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aee5e9280b8a48b6ada6dd7286f51706, State: Initialized, Role: FOLLOWER
I20260619 14:37:28.399766 2017 consensus_queue.cc:260] T 668e4a1a14d6463ea0f4488477fdb5e5 P aee5e9280b8a48b6ada6dd7286f51706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422fcf83e69b42988c71e1a5a7774760" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39197 } } peers { permanent_uuid: "6b28eaa67a5d4711a16af5366ff2b67f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37667 } } peers { permanent_uuid: "aee5e9280b8a48b6ada6dd7286f51706" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42729 } }
I20260619 14:37:28.399984 2017 ts_tablet_manager.cc:1428] T 668e4a1a14d6463ea0f4488477fdb5e5 P aee5e9280b8a48b6ada6dd7286f51706: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:37:28.400173 2024 tablet_replica.cc:439] T 668e4a1a14d6463ea0f4488477fdb5e5 P aee5e9280b8a48b6ada6dd7286f51706: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422fcf83e69b42988c71e1a5a7774760" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39197 } } peers { permanent_uuid: "6b28eaa67a5d4711a16af5366ff2b67f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37667 } } peers { permanent_uuid: "aee5e9280b8a48b6ada6dd7286f51706" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42729 } } }
I20260619 14:37:28.400293 2024 tablet_replica.cc:442] T 668e4a1a14d6463ea0f4488477fdb5e5 P aee5e9280b8a48b6ada6dd7286f51706: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:37:28.556262 2022 raft_consensus.cc:482] T 668e4a1a14d6463ea0f4488477fdb5e5 P 422fcf83e69b42988c71e1a5a7774760 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:37:28.556397 2022 raft_consensus.cc:504] T 668e4a1a14d6463ea0f4488477fdb5e5 P 422fcf83e69b42988c71e1a5a7774760 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422fcf83e69b42988c71e1a5a7774760" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39197 } } peers { permanent_uuid: "6b28eaa67a5d4711a16af5366ff2b67f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37667 } } peers { permanent_uuid: "aee5e9280b8a48b6ada6dd7286f51706" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42729 } }
I20260619 14:37:28.557448 2022 leader_election.cc:288] T 668e4a1a14d6463ea0f4488477fdb5e5 P 422fcf83e69b42988c71e1a5a7774760 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b28eaa67a5d4711a16af5366ff2b67f (127.0.98.2:37667), aee5e9280b8a48b6ada6dd7286f51706 (127.0.98.3:42729)
I20260619 14:37:28.561101 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:28.557753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45329 (local address 127.0.98.2:37667)
0619 14:37:28.557886 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:28.557890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:28.558088 (+ 198us) server_negotiation.cc:407] Connection header received
0619 14:37:28.558192 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:28.558195 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:28.558244 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:28.558302 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:37:28.558600 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:28.559074 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:28.560155 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:28.560540 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:28.560641 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:28.560703 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:28.560924 (+ 221us) server_negotiation.cc:299] Negotiation successful
0619 14:37:28.561000 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 14:37:28.561097 2058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:28.557815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58487 (local address 127.0.98.3:42729)
0619 14:37:28.558026 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:28.558031 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:28.558049 (+ 18us) server_negotiation.cc:407] Connection header received
0619 14:37:28.558231 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:28.558233 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:28.558273 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:28.558353 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:37:28.558568 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:28.559084 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:28.560171 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:28.560557 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:28.560654 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:28.560714 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:28.560895 (+ 181us) server_negotiation.cc:299] Negotiation successful
0619 14:37:28.560964 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":50,"threads_started":1}
I20260619 14:37:28.561110 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:28.557672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37667 (local address 127.0.98.1:45329)
0619 14:37:28.557983 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:28.557997 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:28.558102 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:28.558333 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:28.558336 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:28.558351 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:37:28.558486 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:28.558491 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:28.559177 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:28.559180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:28.560036 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:28.560044 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:28.560667 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:28.560670 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:28.560788 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:28.560830 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 14:37:28.560899 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 14:37:28.560964 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":32,"thread_start_us":82,"threads_started":1}
I20260619 14:37:28.561113 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:28.557705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42729 (local address 127.0.98.1:58487)
0619 14:37:28.557995 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:28.558007 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:28.558123 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:28.558374 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:28.558377 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:28.558393 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:37:28.558472 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:28.558477 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:28.559196 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:28.559200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:28.560037 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:28.560045 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:28.560665 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:28.560668 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:28.560779 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:28.560808 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 14:37:28.560863 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 14:37:28.560923 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":64,"thread_start_us":68,"threads_started":1}
I20260619 14:37:28.561498 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "668e4a1a14d6463ea0f4488477fdb5e5" candidate_uuid: "422fcf83e69b42988c71e1a5a7774760" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b28eaa67a5d4711a16af5366ff2b67f" is_pre_election: true
I20260619 14:37:28.561633 1807 raft_consensus.cc:2454] T 668e4a1a14d6463ea0f4488477fdb5e5 P 6b28eaa67a5d4711a16af5366ff2b67f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 422fcf83e69b42988c71e1a5a7774760 in term 0.
I20260619 14:37:28.561712 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "668e4a1a14d6463ea0f4488477fdb5e5" candidate_uuid: "422fcf83e69b42988c71e1a5a7774760" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aee5e9280b8a48b6ada6dd7286f51706" is_pre_election: true
I20260619 14:37:28.561848 1947 raft_consensus.cc:2454] T 668e4a1a14d6463ea0f4488477fdb5e5 P aee5e9280b8a48b6ada6dd7286f51706 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 422fcf83e69b42988c71e1a5a7774760 in term 0.
I20260619 14:37:28.561997 1599 leader_election.cc:302] T 668e4a1a14d6463ea0f4488477fdb5e5 P 422fcf83e69b42988c71e1a5a7774760 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 422fcf83e69b42988c71e1a5a7774760, 6b28eaa67a5d4711a16af5366ff2b67f; no voters:
I20260619 14:37:28.562256 2022 raft_consensus.cc:2781] T 668e4a1a14d6463ea0f4488477fdb5e5 P 422fcf83e69b42988c71e1a5a7774760 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:37:28.562302 2022 raft_consensus.cc:482] T 668e4a1a14d6463ea0f4488477fdb5e5 P 422fcf83e69b42988c71e1a5a7774760 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:37:28.562326 2022 raft_consensus.cc:3037] T 668e4a1a14d6463ea0f4488477fdb5e5 P 422fcf83e69b42988c71e1a5a7774760 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:28.563252 2022 raft_consensus.cc:504] T 668e4a1a14d6463ea0f4488477fdb5e5 P 422fcf83e69b42988c71e1a5a7774760 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422fcf83e69b42988c71e1a5a7774760" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39197 } } peers { permanent_uuid: "6b28eaa67a5d4711a16af5366ff2b67f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37667 } } peers { permanent_uuid: "aee5e9280b8a48b6ada6dd7286f51706" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42729 } }
I20260619 14:37:28.563555 2022 leader_election.cc:288] T 668e4a1a14d6463ea0f4488477fdb5e5 P 422fcf83e69b42988c71e1a5a7774760 [CANDIDATE]: Term 1 election: Requested vote from peers 6b28eaa67a5d4711a16af5366ff2b67f (127.0.98.2:37667), aee5e9280b8a48b6ada6dd7286f51706 (127.0.98.3:42729)
I20260619 14:37:28.563901 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "668e4a1a14d6463ea0f4488477fdb5e5" candidate_uuid: "422fcf83e69b42988c71e1a5a7774760" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b28eaa67a5d4711a16af5366ff2b67f"
I20260619 14:37:28.563905 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "668e4a1a14d6463ea0f4488477fdb5e5" candidate_uuid: "422fcf83e69b42988c71e1a5a7774760" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aee5e9280b8a48b6ada6dd7286f51706"
I20260619 14:37:28.563998 1807 raft_consensus.cc:3037] T 668e4a1a14d6463ea0f4488477fdb5e5 P 6b28eaa67a5d4711a16af5366ff2b67f [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:28.564002 1947 raft_consensus.cc:3037] T 668e4a1a14d6463ea0f4488477fdb5e5 P aee5e9280b8a48b6ada6dd7286f51706 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:28.564851 1807 raft_consensus.cc:2454] T 668e4a1a14d6463ea0f4488477fdb5e5 P 6b28eaa67a5d4711a16af5366ff2b67f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 422fcf83e69b42988c71e1a5a7774760 in term 1.
I20260619 14:37:28.564880 1947 raft_consensus.cc:2454] T 668e4a1a14d6463ea0f4488477fdb5e5 P aee5e9280b8a48b6ada6dd7286f51706 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 422fcf83e69b42988c71e1a5a7774760 in term 1.
I20260619 14:37:28.565132 1601 leader_election.cc:302] T 668e4a1a14d6463ea0f4488477fdb5e5 P 422fcf83e69b42988c71e1a5a7774760 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 422fcf83e69b42988c71e1a5a7774760, aee5e9280b8a48b6ada6dd7286f51706; no voters:
I20260619 14:37:28.565286 2022 raft_consensus.cc:2781] T 668e4a1a14d6463ea0f4488477fdb5e5 P 422fcf83e69b42988c71e1a5a7774760 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:37:28.565439 2022 raft_consensus.cc:686] T 668e4a1a14d6463ea0f4488477fdb5e5 P 422fcf83e69b42988c71e1a5a7774760 [term 1 LEADER]: Becoming Leader. State: Replica: 422fcf83e69b42988c71e1a5a7774760, State: Running, Role: LEADER
I20260619 14:37:28.565680 2022 consensus_queue.cc:237] T 668e4a1a14d6463ea0f4488477fdb5e5 P 422fcf83e69b42988c71e1a5a7774760 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422fcf83e69b42988c71e1a5a7774760" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39197 } } peers { permanent_uuid: "6b28eaa67a5d4711a16af5366ff2b67f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37667 } } peers { permanent_uuid: "aee5e9280b8a48b6ada6dd7286f51706" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42729 } }
I20260619 14:37:28.566660 2059 tablet_replica.cc:439] T 668e4a1a14d6463ea0f4488477fdb5e5 P 422fcf83e69b42988c71e1a5a7774760: TxnStatusTablet state changed. Reason: New leader 422fcf83e69b42988c71e1a5a7774760. Latest consensus state: current_term: 1 leader_uuid: "422fcf83e69b42988c71e1a5a7774760" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422fcf83e69b42988c71e1a5a7774760" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39197 } } peers { permanent_uuid: "6b28eaa67a5d4711a16af5366ff2b67f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37667 } } peers { permanent_uuid: "aee5e9280b8a48b6ada6dd7286f51706" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42729 } } }
I20260619 14:37:28.566764 2059 tablet_replica.cc:442] T 668e4a1a14d6463ea0f4488477fdb5e5 P 422fcf83e69b42988c71e1a5a7774760: This TxnStatusTablet replica's current role is: LEADER
I20260619 14:37:28.567003 2061 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 14:37:28.567157 2061 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 14:37:28.567749 1508 catalog_manager.cc:5495] T 668e4a1a14d6463ea0f4488477fdb5e5 P 422fcf83e69b42988c71e1a5a7774760 reported cstate change: term changed from 0 to 1, leader changed from <none> to 422fcf83e69b42988c71e1a5a7774760 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "422fcf83e69b42988c71e1a5a7774760" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422fcf83e69b42988c71e1a5a7774760" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b28eaa67a5d4711a16af5366ff2b67f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aee5e9280b8a48b6ada6dd7286f51706" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42729 } health_report { overall_health: UNKNOWN } } }
I20260619 14:37:28.587179 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:28.583736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39197 (local address 127.0.0.1:56230)
0619 14:37:28.583954 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:28.583968 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:28.584073 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:28.584345 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:28.584347 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:28.584357 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:37:28.584454 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:28.584459 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:28.585172 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:28.585176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:28.585785 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:28.585793 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:28.586184 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:28.586188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:28.586284 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:28.586325 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:37:28.586910 (+ 585us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:37:28.586923 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 14:37:28.586985 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 14:37:28.587067 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126}
I20260619 14:37:28.587170 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:28.583801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56230 (local address 127.0.98.1:39197)
0619 14:37:28.583896 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:28.583901 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:28.584022 (+ 121us) server_negotiation.cc:407] Connection header received
0619 14:37:28.584194 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:28.584197 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:28.584245 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:28.584323 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:37:28.584541 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:28.585080 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:28.585892 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:28.586095 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:28.586175 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:28.586451 (+ 276us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:37:28.586825 (+ 374us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:37:28.586920 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:28.586994 (+ 74us) server_negotiation.cc:299] Negotiation successful
0619 14:37:28.587043 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 14:37:28.590147 1947 raft_consensus.cc:1261] T 668e4a1a14d6463ea0f4488477fdb5e5 P aee5e9280b8a48b6ada6dd7286f51706 [term 1 FOLLOWER]: Refusing update from remote peer 422fcf83e69b42988c71e1a5a7774760: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:37:28.590426 1807 raft_consensus.cc:1261] T 668e4a1a14d6463ea0f4488477fdb5e5 P 6b28eaa67a5d4711a16af5366ff2b67f [term 1 FOLLOWER]: Refusing update from remote peer 422fcf83e69b42988c71e1a5a7774760: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:37:28.590549 2059 consensus_queue.cc:1035] T 668e4a1a14d6463ea0f4488477fdb5e5 P 422fcf83e69b42988c71e1a5a7774760 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aee5e9280b8a48b6ada6dd7286f51706" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:37:28.590922 2022 consensus_queue.cc:1035] T 668e4a1a14d6463ea0f4488477fdb5e5 P 422fcf83e69b42988c71e1a5a7774760 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b28eaa67a5d4711a16af5366ff2b67f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:37:28.592095 2024 tablet_replica.cc:439] T 668e4a1a14d6463ea0f4488477fdb5e5 P aee5e9280b8a48b6ada6dd7286f51706: TxnStatusTablet state changed. Reason: New leader 422fcf83e69b42988c71e1a5a7774760. Latest consensus state: current_term: 1 leader_uuid: "422fcf83e69b42988c71e1a5a7774760" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422fcf83e69b42988c71e1a5a7774760" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39197 } } peers { permanent_uuid: "6b28eaa67a5d4711a16af5366ff2b67f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37667 } } peers { permanent_uuid: "aee5e9280b8a48b6ada6dd7286f51706" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42729 } } }
I20260619 14:37:28.592206 2024 tablet_replica.cc:442] T 668e4a1a14d6463ea0f4488477fdb5e5 P aee5e9280b8a48b6ada6dd7286f51706: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:37:28.592530 2026 tablet_replica.cc:439] T 668e4a1a14d6463ea0f4488477fdb5e5 P 6b28eaa67a5d4711a16af5366ff2b67f: TxnStatusTablet state changed. Reason: New leader 422fcf83e69b42988c71e1a5a7774760. Latest consensus state: current_term: 1 leader_uuid: "422fcf83e69b42988c71e1a5a7774760" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422fcf83e69b42988c71e1a5a7774760" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39197 } } peers { permanent_uuid: "6b28eaa67a5d4711a16af5366ff2b67f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37667 } } peers { permanent_uuid: "aee5e9280b8a48b6ada6dd7286f51706" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42729 } } }
I20260619 14:37:28.592631 2026 tablet_replica.cc:442] T 668e4a1a14d6463ea0f4488477fdb5e5 P 6b28eaa67a5d4711a16af5366ff2b67f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:37:28.594241 2022 tablet_replica.cc:439] T 668e4a1a14d6463ea0f4488477fdb5e5 P 422fcf83e69b42988c71e1a5a7774760: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "422fcf83e69b42988c71e1a5a7774760" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422fcf83e69b42988c71e1a5a7774760" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39197 } } peers { permanent_uuid: "6b28eaa67a5d4711a16af5366ff2b67f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37667 } } peers { permanent_uuid: "aee5e9280b8a48b6ada6dd7286f51706" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42729 } } }
I20260619 14:37:28.596637 2022 tablet_replica.cc:442] T 668e4a1a14d6463ea0f4488477fdb5e5 P 422fcf83e69b42988c71e1a5a7774760: This TxnStatusTablet replica's current role is: LEADER
I20260619 14:37:28.596706 2026 tablet_replica.cc:439] T 668e4a1a14d6463ea0f4488477fdb5e5 P 6b28eaa67a5d4711a16af5366ff2b67f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "422fcf83e69b42988c71e1a5a7774760" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422fcf83e69b42988c71e1a5a7774760" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39197 } } peers { permanent_uuid: "6b28eaa67a5d4711a16af5366ff2b67f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37667 } } peers { permanent_uuid: "aee5e9280b8a48b6ada6dd7286f51706" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42729 } } }
I20260619 14:37:28.596172 2059 tablet_replica.cc:439] T 668e4a1a14d6463ea0f4488477fdb5e5 P 422fcf83e69b42988c71e1a5a7774760: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "422fcf83e69b42988c71e1a5a7774760" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422fcf83e69b42988c71e1a5a7774760" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39197 } } peers { permanent_uuid: "6b28eaa67a5d4711a16af5366ff2b67f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37667 } } peers { permanent_uuid: "aee5e9280b8a48b6ada6dd7286f51706" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42729 } } }
I20260619 14:37:28.596824 2059 tablet_replica.cc:442] T 668e4a1a14d6463ea0f4488477fdb5e5 P 422fcf83e69b42988c71e1a5a7774760: This TxnStatusTablet replica's current role is: LEADER
I20260619 14:37:28.596828 2026 tablet_replica.cc:442] T 668e4a1a14d6463ea0f4488477fdb5e5 P 6b28eaa67a5d4711a16af5366ff2b67f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:37:28.598459 2024 tablet_replica.cc:439] T 668e4a1a14d6463ea0f4488477fdb5e5 P aee5e9280b8a48b6ada6dd7286f51706: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "422fcf83e69b42988c71e1a5a7774760" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422fcf83e69b42988c71e1a5a7774760" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39197 } } peers { permanent_uuid: "6b28eaa67a5d4711a16af5366ff2b67f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37667 } } peers { permanent_uuid: "aee5e9280b8a48b6ada6dd7286f51706" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42729 } } }
I20260619 14:37:28.598588 2024 tablet_replica.cc:442] T 668e4a1a14d6463ea0f4488477fdb5e5 P aee5e9280b8a48b6ada6dd7286f51706: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:37:28.604260 2058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:28.600484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59786 (local address 127.0.98.3:42729)
0619 14:37:28.600603 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:28.600608 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:28.600752 (+ 144us) server_negotiation.cc:407] Connection header received
0619 14:37:28.600921 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:28.600924 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:28.600985 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:28.601090 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:37:28.601249 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:28.602017 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:28.602818 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:28.603151 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:28.603241 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:28.603423 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:37:28.603919 (+ 496us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:37:28.604023 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:28.604085 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 14:37:28.604137 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 14:37:28.613366 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:28.610402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49611 (local address 127.0.98.1:39197)
0619 14:37:28.610510 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:28.610515 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:28.610704 (+ 189us) server_negotiation.cc:407] Connection header received
0619 14:37:28.610814 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:28.610817 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:28.610856 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:28.610911 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:37:28.611111 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:28.611656 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:28.612453 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:28.612656 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:28.612772 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:28.613015 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:37:28.613075 (+ 60us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:37:28.613129 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:28.613211 (+ 82us) server_negotiation.cc:299] Negotiation successful
0619 14:37:28.613252 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 14:37:28.613401 2077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:28.610329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39197 (local address 127.0.98.3:49611)
0619 14:37:28.610634 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:28.610644 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:28.610739 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:28.610924 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:28.610927 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:28.610933 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:37:28.611024 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:28.611029 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:28.611750 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:28.611753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:28.612347 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:28.612355 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:28.612775 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:28.612778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:28.612868 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:28.612905 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:37:28.613142 (+ 237us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:37:28.613148 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 14:37:28.613203 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 14:37:28.613280 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":77,"threads_started":1}
I20260619 14:37:28.619233 1807 raft_consensus.cc:1261] T aa540acd584046dea8c034702e8631f5 P 6b28eaa67a5d4711a16af5366ff2b67f [term 1 FOLLOWER]: Refusing update from remote peer aee5e9280b8a48b6ada6dd7286f51706: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:37:28.619395 1667 raft_consensus.cc:1261] T aa540acd584046dea8c034702e8631f5 P 422fcf83e69b42988c71e1a5a7774760 [term 1 FOLLOWER]: Refusing update from remote peer aee5e9280b8a48b6ada6dd7286f51706: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:37:28.619704 2024 consensus_queue.cc:1035] T aa540acd584046dea8c034702e8631f5 P aee5e9280b8a48b6ada6dd7286f51706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b28eaa67a5d4711a16af5366ff2b67f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:37:28.619870 2045 consensus_queue.cc:1035] T aa540acd584046dea8c034702e8631f5 P aee5e9280b8a48b6ada6dd7286f51706 [LEADER]: Connected to new peer: Peer: permanent_uuid: "422fcf83e69b42988c71e1a5a7774760" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:37:28.623653 2067 mvcc.cc:205] Tried to move back new op lower bound from 7298579859940888576 to 7298579858561191936. Current Snapshot: MvccSnapshot[applied={T|T < 7298579859940888576}]
I20260619 14:37:28.641901 2058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:28.637785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39997 (local address 127.0.98.3:42729)
0619 14:37:28.637876 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:28.637880 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:28.638142 (+ 262us) server_negotiation.cc:407] Connection header received
0619 14:37:28.638286 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:28.638290 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:28.638331 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:28.638370 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:37:28.638676 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:28.639424 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:28.640681 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:28.641051 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:28.641115 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:28.641509 (+ 394us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:37:28.641592 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:37:28.641658 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:28.641744 (+ 86us) server_negotiation.cc:299] Negotiation successful
0619 14:37:28.641787 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 14:37:28.641939 2082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:28.637512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42729 (local address 127.0.98.1:39997)
0619 14:37:28.638071 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:28.638084 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:28.638175 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:28.638483 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:28.638485 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:28.638493 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:37:28.638593 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:28.638598 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:28.639530 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:28.639533 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:28.640557 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:28.640567 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:28.641238 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:28.641242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:28.641337 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:28.641380 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:37:28.641670 (+ 290us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:37:28.641675 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 14:37:28.641725 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 14:37:28.641798 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":83,"threads_started":1}
I20260619 14:37:28.671515 1508 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56294:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:37:28.671687 1508 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56294:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:37:28.673777 1508 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 82fc67ef30144ceabdac56aab5b87dc8: Sending DeleteTablet for 3 replicas of tablet aa540acd584046dea8c034702e8631f5
I20260619 14:37:28.674424 1787 tablet_service.cc:1514] Processing DeleteTablet for tablet aa540acd584046dea8c034702e8631f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:37:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48714
I20260619 14:37:28.674603 1927 tablet_service.cc:1514] Processing DeleteTablet for tablet aa540acd584046dea8c034702e8631f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:37:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59774
I20260619 14:37:28.674747 1647 tablet_service.cc:1514] Processing DeleteTablet for tablet aa540acd584046dea8c034702e8631f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:37:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56214
I20260619 14:37:28.675055 2085 tablet_replica.cc:330] T aa540acd584046dea8c034702e8631f5 P 6b28eaa67a5d4711a16af5366ff2b67f: stopping tablet replica
I20260619 14:37:28.675180 2085 raft_consensus.cc:2229] T aa540acd584046dea8c034702e8631f5 P 6b28eaa67a5d4711a16af5366ff2b67f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:37:28.675280 2085 raft_consensus.cc:2258] T aa540acd584046dea8c034702e8631f5 P 6b28eaa67a5d4711a16af5366ff2b67f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:37:28.675958 2085 ts_tablet_manager.cc:1905] T aa540acd584046dea8c034702e8631f5 P 6b28eaa67a5d4711a16af5366ff2b67f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:37:28.676987 2086 tablet_replica.cc:330] T aa540acd584046dea8c034702e8631f5 P aee5e9280b8a48b6ada6dd7286f51706: stopping tablet replica
I20260619 14:37:28.677162 2086 raft_consensus.cc:2229] T aa540acd584046dea8c034702e8631f5 P aee5e9280b8a48b6ada6dd7286f51706 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:37:28.677533 2086 raft_consensus.cc:2258] T aa540acd584046dea8c034702e8631f5 P aee5e9280b8a48b6ada6dd7286f51706 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:37:28.677816 2087 tablet_replica.cc:330] T aa540acd584046dea8c034702e8631f5 P 422fcf83e69b42988c71e1a5a7774760: stopping tablet replica
I20260619 14:37:28.678354 2086 ts_tablet_manager.cc:1905] T aa540acd584046dea8c034702e8631f5 P aee5e9280b8a48b6ada6dd7286f51706: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:37:28.678994 2085 ts_tablet_manager.cc:1918] T aa540acd584046dea8c034702e8631f5 P 6b28eaa67a5d4711a16af5366ff2b67f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:37:28.679077 2085 log.cc:1196] T aa540acd584046dea8c034702e8631f5 P 6b28eaa67a5d4711a16af5366ff2b67f: Deleting WAL directory at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/ts-1/wal/wals/aa540acd584046dea8c034702e8631f5
I20260619 14:37:28.679252 2087 raft_consensus.cc:2229] T aa540acd584046dea8c034702e8631f5 P 422fcf83e69b42988c71e1a5a7774760 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:37:28.679317 2085 ts_tablet_manager.cc:1939] T aa540acd584046dea8c034702e8631f5 P 6b28eaa67a5d4711a16af5366ff2b67f: Deleting consensus metadata
I20260619 14:37:28.679979 1495 catalog_manager.cc:4841] TS 6b28eaa67a5d4711a16af5366ff2b67f (127.0.98.2:37667): tablet aa540acd584046dea8c034702e8631f5 (table test-table [id=eb6ac2fc22134316b3d883279d8cdbc2]) successfully deleted
I20260619 14:37:28.680191 2087 raft_consensus.cc:2258] T aa540acd584046dea8c034702e8631f5 P 422fcf83e69b42988c71e1a5a7774760 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:37:28.681775 2086 ts_tablet_manager.cc:1918] T aa540acd584046dea8c034702e8631f5 P aee5e9280b8a48b6ada6dd7286f51706: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:37:28.681856 2086 log.cc:1196] T aa540acd584046dea8c034702e8631f5 P aee5e9280b8a48b6ada6dd7286f51706: Deleting WAL directory at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/ts-2/wal/wals/aa540acd584046dea8c034702e8631f5
I20260619 14:37:28.682089 2086 ts_tablet_manager.cc:1939] T aa540acd584046dea8c034702e8631f5 P aee5e9280b8a48b6ada6dd7286f51706: Deleting consensus metadata
I20260619 14:37:28.682723 1497 catalog_manager.cc:4841] TS aee5e9280b8a48b6ada6dd7286f51706 (127.0.98.3:42729): tablet aa540acd584046dea8c034702e8631f5 (table test-table [id=eb6ac2fc22134316b3d883279d8cdbc2]) successfully deleted
I20260619 14:37:28.683271 2087 ts_tablet_manager.cc:1905] T aa540acd584046dea8c034702e8631f5 P 422fcf83e69b42988c71e1a5a7774760: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:37:28.685904 2087 ts_tablet_manager.cc:1918] T aa540acd584046dea8c034702e8631f5 P 422fcf83e69b42988c71e1a5a7774760: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:37:28.685981 2087 log.cc:1196] T aa540acd584046dea8c034702e8631f5 P 422fcf83e69b42988c71e1a5a7774760: Deleting WAL directory at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781879837648036-392-0/minicluster-data/ts-0/wal/wals/aa540acd584046dea8c034702e8631f5
I20260619 14:37:28.686214 2087 ts_tablet_manager.cc:1939] T aa540acd584046dea8c034702e8631f5 P 422fcf83e69b42988c71e1a5a7774760: Deleting consensus metadata
I20260619 14:37:28.686906 1497 catalog_manager.cc:4841] TS 422fcf83e69b42988c71e1a5a7774760 (127.0.98.1:39197): tablet aa540acd584046dea8c034702e8631f5 (table test-table [id=eb6ac2fc22134316b3d883279d8cdbc2]) successfully deleted
Jun 19 14:37:28 dist-test-slave-32hj krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879847, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 14:37:28.692852 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:28.676386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56242 (local address 127.0.98.1:39197)
0619 14:37:28.676494 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:28.676498 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:28.681057 (+ 4559us) server_negotiation.cc:407] Connection header received
0619 14:37:28.681292 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:28.681296 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:28.681424 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:28.681478 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:28.682654 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:28.683394 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:28.684643 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:28.685009 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:28.685081 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:28.688974 (+ 3893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:28.688984 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:28.688997 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:28.689024 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:28.690876 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:28.691410 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:28.691415 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:28.691417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:28.691521 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:28.691784 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:28.691787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:28.691789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:28.692084 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:28.692299 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:28.692492 (+ 193us) server_negotiation.cc:299] Negotiation successful
0619 14:37:28.692622 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260619 14:37:28.693249 1707 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56242
I20260619 14:37:28.703428 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:28.694946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56304 (local address 127.0.98.62:43579)
0619 14:37:28.695062 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:28.695066 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:28.695472 (+ 406us) server_negotiation.cc:407] Connection header received
0619 14:37:28.695564 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:28.695566 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:28.695617 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:28.695698 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:28.696412 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:28.697252 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:28.698340 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:28.698675 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:28.698781 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:28.700175 (+ 1394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:28.700179 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:28.700182 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:28.700207 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:28.701812 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:28.702254 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:28.702257 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:28.702259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:28.702354 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:28.702607 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:28.702610 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:28.702612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:28.702819 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:28.702950 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:28.703161 (+ 211us) server_negotiation.cc:299] Negotiation successful
0619 14:37:28.703276 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260619 14:37:28.703783 1508 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56304
I20260619 14:37:28.705185 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 1582
I20260619 14:37:28.712985 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 1722
I20260619 14:37:28.718743 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 1862
I20260619 14:37:28.723500 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 1478
2026-06-19T14:37:28Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5313 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:37:28 dist-test-slave-32hj krb5kdc[2094](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:37:28 dist-test-slave-32hj krb5kdc[2094](info): set up 2 sockets
Jun 19 14:37:28 dist-test-slave-32hj krb5kdc[2094](info): commencing operation
krb5kdc: starting...
W20260619 14:37:29.672293 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/krb5kdc/test-user.keytab.
Jun 19 14:37:29 dist-test-slave-32hj krb5kdc[2094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:37:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:37:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:37:29.780869 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35971
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35971
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:37:29.886936 2107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:37:29.887187 2107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:37:29.887223 2107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:37:29.891131 2107 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:37:29.891187 2107 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 14:37:29.891202 2107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:37:29.891218 2107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:37:29.891235 2107 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:37:29.891263 2107 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:37:29.896440 2107 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35971
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:37:29.897467 2107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:37:29.898439 2107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:37:29.906287 2113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:29.906380 2107 server_base.cc:1034] running on GCE node
W20260619 14:37:29.906356 2115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:29.906298 2112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:29.907936 2107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:37:29.908972 2107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:37:29.910132 2107 hybrid_clock.cc:648] HybridClock initialized: now 1781879849910112 us; error 33 us; skew 500 ppm
Jun 19 14:37:29 dist-test-slave-32hj krb5kdc[2094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879849, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:37:29.912782 2107 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:37:29.913600 2107 webserver.cc:460] Webserver started at http://127.0.98.62:43397/ using document root <none> and password file <none>
I20260619 14:37:29.914101 2107 fs_manager.cc:359] Metadata directory not provided
I20260619 14:37:29.914160 2107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:37:29.914341 2107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:37:29.916297 2107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/master-0/data/instance:
uuid: "96890e15464d40b1a3c6643bcb0abd9a"
format_stamp: "Formatted at 2026-06-19 14:37:29 on dist-test-slave-32hj"
server_key: "c18b10d384ad714cf97ba5de10fc3fbf"
server_key_iv: "fc88e4335b1cac7448ce1b7071bd876e"
server_key_version: "encryptionkey@0"
I20260619 14:37:29.916859 2107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/master-0/wal/instance:
uuid: "96890e15464d40b1a3c6643bcb0abd9a"
format_stamp: "Formatted at 2026-06-19 14:37:29 on dist-test-slave-32hj"
server_key: "c18b10d384ad714cf97ba5de10fc3fbf"
server_key_iv: "fc88e4335b1cac7448ce1b7071bd876e"
server_key_version: "encryptionkey@0"
I20260619 14:37:29.920224 2107 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:37:29.922456 2122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:29.923614 2107 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:37:29.923736 2107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/master-0/wal
uuid: "96890e15464d40b1a3c6643bcb0abd9a"
format_stamp: "Formatted at 2026-06-19 14:37:29 on dist-test-slave-32hj"
server_key: "c18b10d384ad714cf97ba5de10fc3fbf"
server_key_iv: "fc88e4335b1cac7448ce1b7071bd876e"
server_key_version: "encryptionkey@0"
I20260619 14:37:29.923828 2107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:37:29.943230 2107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:37:29.946200 2107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:37:29.946363 2107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:37:29.953558 2107 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35971
I20260619 14:37:29.953572 2184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35971 every 8 connection(s)
I20260619 14:37:29.954447 2107 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:37:29.956084 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 2107
I20260619 14:37:29.956171 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/master-0/wal/instance
I20260619 14:37:29.956418 392 external_mini_cluster.cc:1428] Setting key eba13af9ae875b66d3518ff43ad61595
I20260619 14:37:29.957227 2185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:29.963256 2185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96890e15464d40b1a3c6643bcb0abd9a: Bootstrap starting.
Jun 19 14:37:29 dist-test-slave-32hj krb5kdc[2094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:37:29.965808 2185 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96890e15464d40b1a3c6643bcb0abd9a: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:29.967001 2185 log.cc:824] T 00000000000000000000000000000000 P 96890e15464d40b1a3c6643bcb0abd9a: Log is configured to *not* fsync() on all Append() calls
I20260619 14:37:29.969120 2185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96890e15464d40b1a3c6643bcb0abd9a: No bootstrap required, opened a new log
I20260619 14:37:29.970484 2188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:29.957716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45246 (local address 127.0.98.62:35971)
0619 14:37:29.958095 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:29.958111 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:29.958146 (+ 35us) server_negotiation.cc:407] Connection header received
0619 14:37:29.958760 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:29.958776 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:29.959087 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:29.959353 (+ 266us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:29.960212 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:29.961264 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:29.962408 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:29.962873 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:29.962982 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:29.965586 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:29.965592 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:29.965607 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:29.965636 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:29.967492 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:29.967933 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:29.967938 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:29.967944 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:29.968081 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:29.968389 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:29.968392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:29.968393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:29.968875 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:29.969055 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:29.969306 (+ 251us) server_negotiation.cc:299] Negotiation successful
0619 14:37:29.969591 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":105,"threads_started":1}
I20260619 14:37:29.972601 2185 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96890e15464d40b1a3c6643bcb0abd9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96890e15464d40b1a3c6643bcb0abd9a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35971 } }
I20260619 14:37:29.972884 2185 raft_consensus.cc:374] T 00000000000000000000000000000000 P 96890e15464d40b1a3c6643bcb0abd9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:29.972929 2185 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96890e15464d40b1a3c6643bcb0abd9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96890e15464d40b1a3c6643bcb0abd9a, State: Initialized, Role: FOLLOWER
I20260619 14:37:29.973323 2185 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96890e15464d40b1a3c6643bcb0abd9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96890e15464d40b1a3c6643bcb0abd9a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35971 } }
I20260619 14:37:29.973438 2185 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96890e15464d40b1a3c6643bcb0abd9a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:37:29.973480 2185 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96890e15464d40b1a3c6643bcb0abd9a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:37:29.973546 2185 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96890e15464d40b1a3c6643bcb0abd9a [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:29.974367 2185 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96890e15464d40b1a3c6643bcb0abd9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96890e15464d40b1a3c6643bcb0abd9a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35971 } }
I20260619 14:37:29.974644 2185 leader_election.cc:302] T 00000000000000000000000000000000 P 96890e15464d40b1a3c6643bcb0abd9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96890e15464d40b1a3c6643bcb0abd9a; no voters:
I20260619 14:37:29.974920 2185 leader_election.cc:288] T 00000000000000000000000000000000 P 96890e15464d40b1a3c6643bcb0abd9a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:37:29.975023 2190 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96890e15464d40b1a3c6643bcb0abd9a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:37:29.975206 2190 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96890e15464d40b1a3c6643bcb0abd9a [term 1 LEADER]: Becoming Leader. State: Replica: 96890e15464d40b1a3c6643bcb0abd9a, State: Running, Role: LEADER
I20260619 14:37:29.975502 2190 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96890e15464d40b1a3c6643bcb0abd9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96890e15464d40b1a3c6643bcb0abd9a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35971 } }
I20260619 14:37:29.976007 2185 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96890e15464d40b1a3c6643bcb0abd9a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:37:29.976533 2190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96890e15464d40b1a3c6643bcb0abd9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96890e15464d40b1a3c6643bcb0abd9a. Latest consensus state: current_term: 1 leader_uuid: "96890e15464d40b1a3c6643bcb0abd9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96890e15464d40b1a3c6643bcb0abd9a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35971 } } }
I20260619 14:37:29.976768 2190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96890e15464d40b1a3c6643bcb0abd9a [sys.catalog]: This master's current role is: LEADER
I20260619 14:37:29.976960 2192 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96890e15464d40b1a3c6643bcb0abd9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "96890e15464d40b1a3c6643bcb0abd9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96890e15464d40b1a3c6643bcb0abd9a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35971 } } }
I20260619 14:37:29.977064 2192 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96890e15464d40b1a3c6643bcb0abd9a [sys.catalog]: This master's current role is: LEADER
I20260619 14:37:29.977147 2197 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:37:29.980085 2197 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:37:29.985414 2197 catalog_manager.cc:1347] Generated new cluster ID: 31d87afd67764d12958573c9d628be95
I20260619 14:37:29.985517 2197 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:37:30.021737 2197 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:37:30.023023 2197 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:37:30.031307 2197 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 96890e15464d40b1a3c6643bcb0abd9a: Generated new TSK 0
I20260619 14:37:30.032124 2197 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 14:37:30.067651 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35971
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:37:30.173359 2211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:37:30.173659 2211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:37:30.173718 2211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:37:30.177798 2211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:37:30.177845 2211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:37:30.177891 2211 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 14:37:30.177917 2211 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 14:37:30.177969 2211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:37:30.183400 2211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:37:30.184410 2211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:37:30.185400 2211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:37:30.194108 2219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:30.194109 2217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:30.194110 2216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:30.194278 2211 server_base.cc:1034] running on GCE node
I20260619 14:37:30.194712 2211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:37:30.195290 2211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:37:30.196451 2211 hybrid_clock.cc:648] HybridClock initialized: now 1781879850196424 us; error 41 us; skew 500 ppm
Jun 19 14:37:30 dist-test-slave-32hj krb5kdc[2094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879850, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:37:30.198988 2211 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:37:30.199767 2211 webserver.cc:460] Webserver started at http://127.0.98.1:44613/ using document root <none> and password file <none>
I20260619 14:37:30.200244 2211 fs_manager.cc:359] Metadata directory not provided
I20260619 14:37:30.200300 2211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:37:30.200451 2211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:37:30.202361 2211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/ts-0/data/instance:
uuid: "74f0ef49736744ceaccad487bd25d1cf"
format_stamp: "Formatted at 2026-06-19 14:37:30 on dist-test-slave-32hj"
server_key: "389890026554be7e906f60a2b2119e69"
server_key_iv: "604b1c39da1df68b4be08de0d6620656"
server_key_version: "encryptionkey@0"
I20260619 14:37:30.202884 2211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/ts-0/wal/instance:
uuid: "74f0ef49736744ceaccad487bd25d1cf"
format_stamp: "Formatted at 2026-06-19 14:37:30 on dist-test-slave-32hj"
server_key: "389890026554be7e906f60a2b2119e69"
server_key_iv: "604b1c39da1df68b4be08de0d6620656"
server_key_version: "encryptionkey@0"
I20260619 14:37:30.206269 2211 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:37:30.208504 2226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:30.209686 2211 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 14:37:30.209779 2211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/ts-0/wal
uuid: "74f0ef49736744ceaccad487bd25d1cf"
format_stamp: "Formatted at 2026-06-19 14:37:30 on dist-test-slave-32hj"
server_key: "389890026554be7e906f60a2b2119e69"
server_key_iv: "604b1c39da1df68b4be08de0d6620656"
server_key_version: "encryptionkey@0"
I20260619 14:37:30.209856 2211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:37:30.220245 2211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:37:30.222976 2211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:37:30.223104 2211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:37:30.224334 2211 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:37:30.224391 2211 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:30.224440 2211 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:37:30.224472 2211 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 14:37:30 dist-test-slave-32hj krb5kdc[2094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879850, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:37:30.234099 2211 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33225
I20260619 14:37:30.234293 2341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33225 every 8 connection(s)
I20260619 14:37:30.235054 2211 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:37:30.236745 2188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:30.224981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50353 (local address 127.0.98.62:35971)
0619 14:37:30.225121 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:30.225125 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:30.226013 (+ 888us) server_negotiation.cc:407] Connection header received
0619 14:37:30.226949 (+ 936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:30.226952 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:30.227007 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:30.227096 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:30.228406 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:30.229075 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:30.230201 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:30.230434 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:30.230520 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:30.233305 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:30.233309 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:30.233312 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:30.233338 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:30.234965 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:30.235468 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:30.235472 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:30.235474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:30.235561 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:30.235897 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:30.235900 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:30.235901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:30.236065 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:30.236153 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:30.236518 (+ 365us) server_negotiation.cc:299] Negotiation successful
0619 14:37:30.236608 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 14:37:30.237459 2235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:30.225450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35971 (local address 127.0.98.1:50353)
0619 14:37:30.225876 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:30.225915 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:30.226753 (+ 838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:30.227386 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:30.227395 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:30.227814 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:30.228269 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:30.228285 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:30.229187 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:30.229190 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:30.230089 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:30.230098 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:30.230580 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:30.230584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:30.230764 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:30.231306 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:30.231323 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:30.233165 (+ 1842us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:30.235071 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:30.235075 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:30.235105 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:30.235355 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:30.235662 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:30.235665 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:30.235667 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:30.235815 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:30.236145 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:30.236150 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:30.236350 (+ 200us) client_negotiation.cc:769] Sending connection context
0619 14:37:30.236486 (+ 136us) client_negotiation.cc:240] Negotiation successful
0619 14:37:30.236697 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":118,"threads_started":1}
I20260619 14:37:30.238503 2342 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35971
I20260619 14:37:30.238695 2342 heartbeater.cc:464] Registering TS with master...
I20260619 14:37:30.239193 2342 heartbeater.cc:511] Master 127.0.98.62:35971 requested a full tablet report, sending...
I20260619 14:37:30.240566 2138 ts_manager.cc:194] Registered new tserver with Master: 74f0ef49736744ceaccad487bd25d1cf (127.0.98.1:33225)
I20260619 14:37:30.241981 2138 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50353
I20260619 14:37:30.243156 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 2211
I20260619 14:37:30.243256 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:37:30.243538 392 external_mini_cluster.cc:1428] Setting key 12b2ba284f7e9454ba454a88983bb443
I20260619 14:37:30.247081 2348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:30.240511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35971 (local address 127.0.98.1:44431)
0619 14:37:30.240755 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:30.240769 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:30.240857 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:30.241088 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:30.241090 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:30.241260 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:30.241346 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:30.241351 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:30.242271 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:30.242274 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:30.242875 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:30.242884 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:30.243488 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:30.243492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:30.243579 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:30.244022 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:30.244036 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:30.244354 (+ 318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:30.245845 (+ 1491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:30.245848 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:30.245851 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:30.246075 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:30.246337 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:30.246340 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:30.246341 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:30.246408 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:30.246726 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:30.246728 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:30.246791 (+ 63us) client_negotiation.cc:769] Sending connection context
0619 14:37:30.246859 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 14:37:30.246949 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":74,"threads_started":1}
I20260619 14:37:30.247167 2188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:30.240587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44431 (local address 127.0.98.62:35971)
0619 14:37:30.240699 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:30.240703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:30.240834 (+ 131us) server_negotiation.cc:407] Connection header received
0619 14:37:30.240935 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:30.240937 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:30.240973 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:30.241038 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:30.241450 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:30.242166 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:30.242998 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:30.243291 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:30.243382 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:30.244464 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:30.244467 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:30.244470 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:30.244493 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:30.245747 (+ 1254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:30.246163 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:30.246166 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:30.246168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:30.246260 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:30.246473 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:30.246475 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:30.246477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:30.246655 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:30.246742 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:30.246928 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 14:37:30.247033 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 14:37:30.265161 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35971
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:37:30.368410 2351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:37:30.368628 2351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:37:30.368669 2351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:37:30.372475 2351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:37:30.372520 2351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:37:30.372557 2351 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 14:37:30.372575 2351 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 14:37:30.372604 2351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:37:30.377595 2351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:37:30.378437 2351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:37:30.379302 2351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:37:30.387936 2359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:30.388019 2357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:30.388054 2351 server_base.cc:1034] running on GCE node
W20260619 14:37:30.388021 2356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:30.388532 2351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:37:30.389163 2351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:37:30.390321 2351 hybrid_clock.cc:648] HybridClock initialized: now 1781879850390305 us; error 30 us; skew 500 ppm
Jun 19 14:37:30 dist-test-slave-32hj krb5kdc[2094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879850, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:37:30.393021 2351 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:37:30.393816 2351 webserver.cc:460] Webserver started at http://127.0.98.2:33909/ using document root <none> and password file <none>
I20260619 14:37:30.394313 2351 fs_manager.cc:359] Metadata directory not provided
I20260619 14:37:30.394369 2351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:37:30.394554 2351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:37:30.396430 2351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/ts-1/data/instance:
uuid: "ff28141737b8451a813f4524e704bbe7"
format_stamp: "Formatted at 2026-06-19 14:37:30 on dist-test-slave-32hj"
server_key: "f8b5ea7bbbcf0df106d661d379961dc7"
server_key_iv: "a85403995ff526c2be7226a490683034"
server_key_version: "encryptionkey@0"
I20260619 14:37:30.396961 2351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ff28141737b8451a813f4524e704bbe7"
format_stamp: "Formatted at 2026-06-19 14:37:30 on dist-test-slave-32hj"
server_key: "f8b5ea7bbbcf0df106d661d379961dc7"
server_key_iv: "a85403995ff526c2be7226a490683034"
server_key_version: "encryptionkey@0"
I20260619 14:37:30.400327 2351 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 14:37:30.402498 2366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:30.403571 2351 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:37:30.403664 2351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/ts-1/wal
uuid: "ff28141737b8451a813f4524e704bbe7"
format_stamp: "Formatted at 2026-06-19 14:37:30 on dist-test-slave-32hj"
server_key: "f8b5ea7bbbcf0df106d661d379961dc7"
server_key_iv: "a85403995ff526c2be7226a490683034"
server_key_version: "encryptionkey@0"
I20260619 14:37:30.403764 2351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:37:30.421223 2351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:37:30.424118 2351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:37:30.424276 2351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:37:30.425604 2351 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:37:30.425648 2351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:30.425688 2351 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:37:30.425724 2351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 14:37:30 dist-test-slave-32hj krb5kdc[2094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879850, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:37:30.435415 2351 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42505
I20260619 14:37:30.435581 2481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42505 every 8 connection(s)
I20260619 14:37:30.436319 2351 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:37:30.437969 2188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:30.426323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33555 (local address 127.0.98.62:35971)
0619 14:37:30.426475 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:30.426480 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:30.427226 (+ 746us) server_negotiation.cc:407] Connection header received
0619 14:37:30.428303 (+ 1077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:30.428306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:30.428361 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:30.428416 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:30.429783 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:30.430355 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:30.431490 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:30.431700 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:30.431796 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:30.434465 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:30.434470 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:30.434473 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:30.434501 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:30.435914 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:30.436423 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:30.436426 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:30.436428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:30.436505 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:30.436883 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:30.436887 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:30.436888 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:30.437096 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:30.437184 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:30.437694 (+ 510us) server_negotiation.cc:299] Negotiation successful
0619 14:37:30.437817 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 14:37:30.438618 2375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:30.426657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35971 (local address 127.0.98.2:33555)
0619 14:37:30.427090 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:30.427137 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:30.428060 (+ 923us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:30.428823 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:30.428830 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:30.429129 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:30.429620 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:30.429634 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:30.430477 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:30.430480 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:30.431378 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:30.431387 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:30.431849 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:30.431853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:30.432013 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:30.432518 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:30.432538 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:30.434305 (+ 1767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:30.436011 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:30.436017 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:30.436029 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:30.436336 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:30.436586 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:30.436589 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:30.436591 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:30.436787 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:30.437190 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:30.437196 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:30.437468 (+ 272us) client_negotiation.cc:769] Sending connection context
0619 14:37:30.437613 (+ 145us) client_negotiation.cc:240] Negotiation successful
0619 14:37:30.437887 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":101,"threads_started":1}
I20260619 14:37:30.439651 2482 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35971
I20260619 14:37:30.439802 2482 heartbeater.cc:464] Registering TS with master...
I20260619 14:37:30.440011 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 2351
I20260619 14:37:30.440116 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:37:30.440271 2482 heartbeater.cc:511] Master 127.0.98.62:35971 requested a full tablet report, sending...
I20260619 14:37:30.440425 392 external_mini_cluster.cc:1428] Setting key d29fc05191e527db2cfc4bf953bc37ed
I20260619 14:37:30.441238 2139 ts_manager.cc:194] Registered new tserver with Master: ff28141737b8451a813f4524e704bbe7 (127.0.98.2:42505)
I20260619 14:37:30.441902 2139 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33555
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 14:37:30.449352 2488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:30.441962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35971 (local address 127.0.98.2:47547)
0619 14:37:30.442244 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:30.442257 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:30.442333 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:30.443526 (+ 1193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:30.443529 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:30.443692 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:30.443785 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:30.443790 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:30.444528 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:30.444531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:30.445190 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:30.445199 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:30.445664 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:30.445667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:30.445751 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:30.446186 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:30.446195 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:30.446491 (+ 296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:30.447907 (+ 1416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:30.447910 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:30.447912 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:30.448086 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:30.448391 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:30.448393 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:30.448394 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:30.448467 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:30.448926 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:30.448928 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:30.448997 (+ 69us) client_negotiation.cc:769] Sending connection context
0619 14:37:30.449073 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 14:37:30.449194 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":79,"threads_started":1}
I20260619 14:37:30.449411 2188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:30.443242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47547 (local address 127.0.98.62:35971)
0619 14:37:30.443349 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:30.443352 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:30.443364 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:37:30.443394 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:30.443397 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:30.443434 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:30.443494 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:30.443884 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:30.444418 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:30.445295 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:30.445568 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:30.445652 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:30.446608 (+ 956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:30.446612 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:30.446614 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:30.446636 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:30.447814 (+ 1178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:30.448191 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:30.448195 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:30.448197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:30.448296 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:30.448551 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:30.448555 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:30.448557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:30.448841 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:30.448940 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:30.449149 (+ 209us) server_negotiation.cc:299] Negotiation successful
0619 14:37:30.449269 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 14:37:30.463368 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35971
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:37:30.568830 2491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:37:30.569087 2491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:37:30.569151 2491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:37:30.573096 2491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:37:30.573156 2491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:37:30.573218 2491 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 14:37:30.573241 2491 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 14:37:30.573287 2491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:37:30.578552 2491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:37:30.579536 2491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:37:30.580497 2491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:37:30.589260 2499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:30.589313 2497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:30.589268 2496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:30.589566 2491 server_base.cc:1034] running on GCE node
I20260619 14:37:30.589932 2491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:37:30.590515 2491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:37:30.591678 2491 hybrid_clock.cc:648] HybridClock initialized: now 1781879850591656 us; error 37 us; skew 500 ppm
Jun 19 14:37:30 dist-test-slave-32hj krb5kdc[2094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879850, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:37:30.594296 2491 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:37:30.595088 2491 webserver.cc:460] Webserver started at http://127.0.98.3:33325/ using document root <none> and password file <none>
I20260619 14:37:30.595604 2491 fs_manager.cc:359] Metadata directory not provided
I20260619 14:37:30.595669 2491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:37:30.595845 2491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:37:30.597826 2491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/ts-2/data/instance:
uuid: "02b0bac5d7944fb982708b589db96867"
format_stamp: "Formatted at 2026-06-19 14:37:30 on dist-test-slave-32hj"
server_key: "1175dc103dce8f4dd2982074154d4aff"
server_key_iv: "714d2c6ba943b2127bcbdf431057e51f"
server_key_version: "encryptionkey@0"
I20260619 14:37:30.598372 2491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/ts-2/wal/instance:
uuid: "02b0bac5d7944fb982708b589db96867"
format_stamp: "Formatted at 2026-06-19 14:37:30 on dist-test-slave-32hj"
server_key: "1175dc103dce8f4dd2982074154d4aff"
server_key_iv: "714d2c6ba943b2127bcbdf431057e51f"
server_key_version: "encryptionkey@0"
I20260619 14:37:30.601815 2491 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260619 14:37:30.604024 2506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:30.605185 2491 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:37:30.605294 2491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/ts-2/wal
uuid: "02b0bac5d7944fb982708b589db96867"
format_stamp: "Formatted at 2026-06-19 14:37:30 on dist-test-slave-32hj"
server_key: "1175dc103dce8f4dd2982074154d4aff"
server_key_iv: "714d2c6ba943b2127bcbdf431057e51f"
server_key_version: "encryptionkey@0"
I20260619 14:37:30.605379 2491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:37:30.633391 2491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:37:30.636279 2491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:37:30.636443 2491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:37:30.637818 2491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:37:30.637864 2491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:30.637915 2491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:37:30.637943 2491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 14:37:30 dist-test-slave-32hj krb5kdc[2094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879850, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:37:30.647979 2491 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33147
I20260619 14:37:30.648032 2621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33147 every 8 connection(s)
I20260619 14:37:30.649034 2491 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:37:30.650733 2188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:30.638691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34573 (local address 127.0.98.62:35971)
0619 14:37:30.638800 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:30.638803 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:30.639835 (+ 1032us) server_negotiation.cc:407] Connection header received
0619 14:37:30.641014 (+ 1179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:30.641019 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:30.641083 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:30.641179 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:30.642408 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:30.642969 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:30.644192 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:30.644465 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:30.644557 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:30.647204 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:30.647208 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:30.647210 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:30.647236 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:30.648837 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:30.649313 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:30.649316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:30.649318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:30.649417 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:30.649826 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:30.649830 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:30.649831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:30.650041 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:30.650127 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:30.650524 (+ 397us) server_negotiation.cc:299] Negotiation successful
0619 14:37:30.650618 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 14:37:30.651438 2515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:30.639220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35971 (local address 127.0.98.3:34573)
0619 14:37:30.639692 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:30.639732 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:30.640681 (+ 949us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:30.641358 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:30.641367 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:30.641704 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:30.642264 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:30.642280 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:30.643106 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:30.643109 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:30.644076 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:30.644086 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:30.644627 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:30.644632 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:30.644820 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:30.645393 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:30.645412 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:30.647039 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:30.648936 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:30.648940 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:30.648951 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:30.649217 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:30.649525 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:30.649529 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:30.649531 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:30.649714 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:30.650121 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:30.650125 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:30.650341 (+ 216us) client_negotiation.cc:769] Sending connection context
0619 14:37:30.650486 (+ 145us) client_negotiation.cc:240] Negotiation successful
0619 14:37:30.650720 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":128,"threads_started":1}
I20260619 14:37:30.652493 2622 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35971
I20260619 14:37:30.652617 2622 heartbeater.cc:464] Registering TS with master...
I20260619 14:37:30.653110 2622 heartbeater.cc:511] Master 127.0.98.62:35971 requested a full tablet report, sending...
I20260619 14:37:30.654124 2138 ts_manager.cc:194] Registered new tserver with Master: 02b0bac5d7944fb982708b589db96867 (127.0.98.3:33147)
I20260619 14:37:30.654845 2138 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34573
I20260619 14:37:30.658813 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 2491
I20260619 14:37:30.658914 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:37:30.659179 392 external_mini_cluster.cc:1428] Setting key 3b5ff63a17e4a567f8b20a5e3f6760d5
I20260619 14:37:30.660708 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:30.654355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35971 (local address 127.0.98.3:51359)
0619 14:37:30.654581 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:30.654596 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:30.654689 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:30.654956 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:30.654959 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:30.655155 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:30.655244 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:30.655249 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:30.655951 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:30.655955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:30.656498 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:30.656505 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:30.656972 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:30.656976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:30.657059 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:30.657456 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:30.657466 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:30.657746 (+ 280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:30.659353 (+ 1607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:30.659355 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:30.659357 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:30.659585 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:30.659861 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:30.659864 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:30.659866 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:30.659959 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:30.660313 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:30.660316 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:30.660392 (+ 76us) client_negotiation.cc:769] Sending connection context
0619 14:37:30.660472 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 14:37:30.660570 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":75,"threads_started":1}
I20260619 14:37:30.660822 2188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:30.654349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51359 (local address 127.0.98.62:35971)
0619 14:37:30.654466 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:30.654469 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:30.654661 (+ 192us) server_negotiation.cc:407] Connection header received
0619 14:37:30.654810 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:30.654813 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:30.654850 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:30.654928 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:30.655348 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:30.655837 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:30.656598 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:30.656870 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:30.656951 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:30.657864 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:30.657868 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:30.657870 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:30.657893 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:30.659257 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:30.659670 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:30.659673 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:30.659675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:30.659775 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:30.660061 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:30.660064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:30.660066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:30.660233 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:30.660323 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:30.660548 (+ 225us) server_negotiation.cc:299] Negotiation successful
0619 14:37:30.660656 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 14:37:30.661329 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 14:37:30 dist-test-slave-32hj krb5kdc[2094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 14:37:30 dist-test-slave-32hj krb5kdc[2094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:37:30.684067 2188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:30.676589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45248 (local address 127.0.98.62:35971)
0619 14:37:30.676706 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:30.676710 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:30.676735 (+ 25us) server_negotiation.cc:407] Connection header received
0619 14:37:30.676774 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:30.676776 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:30.676811 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:30.676872 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:30.677531 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:30.678052 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:30.678852 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:30.679075 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:30.679161 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:30.681420 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:30.681424 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:30.681427 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:30.681453 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:30.682560 (+ 1107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:30.682928 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:30.682931 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:30.682933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:30.683008 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:30.683272 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:30.683276 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:30.683277 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:30.683454 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:30.683569 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:30.683794 (+ 225us) server_negotiation.cc:299] Negotiation successful
0619 14:37:30.683909 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 14:37:30.686545 2138 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45248:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:37:30.689502 2138 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:37:30.702322 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:30.697368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53802 (local address 127.0.98.1:33225)
0619 14:37:30.698019 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:30.698031 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:30.698055 (+ 24us) server_negotiation.cc:407] Connection header received
0619 14:37:30.698130 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:30.698135 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:30.698337 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:30.698834 (+ 497us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:37:30.698957 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:30.699648 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:30.701038 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:30.701714 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:30.701806 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:30.701963 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:30.702090 (+ 127us) server_negotiation.cc:299] Negotiation successful
0619 14:37:30.702180 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":94,"threads_started":1}
I20260619 14:37:30.702313 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:30.696845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33225 (local address 127.0.0.1:53802)
0619 14:37:30.697509 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:30.697558 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:30.697684 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:30.698504 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:30.698510 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:30.698545 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:37:30.698687 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:30.698695 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:30.699748 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:30.699752 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:30.700936 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:30.700946 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:30.701818 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:30.701821 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:30.701920 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:30.701957 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 14:37:30.702059 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 14:37:30.702175 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"mutex_wait_us":3,"spinlock_wait_cycles":6528,"thread_start_us":76,"threads_started":1}
I20260619 14:37:30.702708 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:30.697365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42505 (local address 127.0.0.1:33156)
0619 14:37:30.697887 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:30.697899 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:30.697969 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:30.698592 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:30.698595 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:30.698613 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:37:30.698759 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:30.698766 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:30.700189 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:30.700192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:30.701263 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:30.701270 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:30.702304 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:30.702308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:30.702444 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:30.702464 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 14:37:30.702518 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 14:37:30.702583 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"mutex_wait_us":104,"thread_start_us":90,"threads_started":1}
I20260619 14:37:30.702945 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:30.696766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33147 (local address 127.0.0.1:45340)
0619 14:37:30.697509 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:30.697558 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:30.697684 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:30.698656 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:30.698659 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:30.698671 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:37:30.698824 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:30.698832 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:30.700228 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:30.700231 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:30.701332 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:30.701341 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:30.702304 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:30.702308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:30.702669 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:30.702693 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 14:37:30.702759 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 14:37:30.702823 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"mutex_wait_us":259,"spinlock_wait_cycles":4096,"thread_start_us":65,"threads_started":1}
I20260619 14:37:30.703333 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:30.696923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45340 (local address 127.0.98.3:33147)
0619 14:37:30.698184 (+ 1261us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:30.698208 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:30.698230 (+ 22us) server_negotiation.cc:407] Connection header received
0619 14:37:30.698302 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:30.698306 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:30.698509 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:30.698800 (+ 291us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:37:30.698930 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:30.700072 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:30.701458 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:30.702191 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:30.702428 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:30.702597 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:30.703094 (+ 497us) server_negotiation.cc:299] Negotiation successful
0619 14:37:30.703190 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1163,"thread_start_us":97,"threads_started":1}
I20260619 14:37:30.703428 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:30.697755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33156 (local address 127.0.98.2:42505)
0619 14:37:30.698215 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:30.698224 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:30.698243 (+ 19us) server_negotiation.cc:407] Connection header received
0619 14:37:30.698302 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:30.698306 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:30.698452 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:30.698615 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:37:30.698893 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:30.700087 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:30.701365 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:30.702165 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:30.703005 (+ 840us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:30.703172 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:30.703217 (+ 45us) server_negotiation.cc:299] Negotiation successful
0619 14:37:30.703298 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":106,"threads_started":1}
I20260619 14:37:30.705721 2556 tablet_service.cc:1467] Processing CreateTablet for tablet a625d7d4cd8549e8ad1ecffabbcf5913 (DEFAULT_TABLE table=test-table [id=b4c39e646d184870a0c64ac613f9149e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:37:30.705559 2276 tablet_service.cc:1467] Processing CreateTablet for tablet a625d7d4cd8549e8ad1ecffabbcf5913 (DEFAULT_TABLE table=test-table [id=b4c39e646d184870a0c64ac613f9149e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:37:30.706041 2416 tablet_service.cc:1467] Processing CreateTablet for tablet a625d7d4cd8549e8ad1ecffabbcf5913 (DEFAULT_TABLE table=test-table [id=b4c39e646d184870a0c64ac613f9149e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:37:30.706640 2556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a625d7d4cd8549e8ad1ecffabbcf5913. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:30.706673 2276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a625d7d4cd8549e8ad1ecffabbcf5913. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:30.706866 2416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a625d7d4cd8549e8ad1ecffabbcf5913. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:30.713819 2645 tablet_bootstrap.cc:492] T a625d7d4cd8549e8ad1ecffabbcf5913 P 02b0bac5d7944fb982708b589db96867: Bootstrap starting.
I20260619 14:37:30.714370 2646 tablet_bootstrap.cc:492] T a625d7d4cd8549e8ad1ecffabbcf5913 P 74f0ef49736744ceaccad487bd25d1cf: Bootstrap starting.
I20260619 14:37:30.714674 2647 tablet_bootstrap.cc:492] T a625d7d4cd8549e8ad1ecffabbcf5913 P ff28141737b8451a813f4524e704bbe7: Bootstrap starting.
I20260619 14:37:30.716236 2647 tablet_bootstrap.cc:654] T a625d7d4cd8549e8ad1ecffabbcf5913 P ff28141737b8451a813f4524e704bbe7: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:30.716248 2645 tablet_bootstrap.cc:654] T a625d7d4cd8549e8ad1ecffabbcf5913 P 02b0bac5d7944fb982708b589db96867: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:30.716248 2646 tablet_bootstrap.cc:654] T a625d7d4cd8549e8ad1ecffabbcf5913 P 74f0ef49736744ceaccad487bd25d1cf: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:30.716967 2647 log.cc:824] T a625d7d4cd8549e8ad1ecffabbcf5913 P ff28141737b8451a813f4524e704bbe7: Log is configured to *not* fsync() on all Append() calls
I20260619 14:37:30.717039 2645 log.cc:824] T a625d7d4cd8549e8ad1ecffabbcf5913 P 02b0bac5d7944fb982708b589db96867: Log is configured to *not* fsync() on all Append() calls
I20260619 14:37:30.717041 2646 log.cc:824] T a625d7d4cd8549e8ad1ecffabbcf5913 P 74f0ef49736744ceaccad487bd25d1cf: Log is configured to *not* fsync() on all Append() calls
I20260619 14:37:30.718675 2647 tablet_bootstrap.cc:492] T a625d7d4cd8549e8ad1ecffabbcf5913 P ff28141737b8451a813f4524e704bbe7: No bootstrap required, opened a new log
I20260619 14:37:30.718683 2646 tablet_bootstrap.cc:492] T a625d7d4cd8549e8ad1ecffabbcf5913 P 74f0ef49736744ceaccad487bd25d1cf: No bootstrap required, opened a new log
I20260619 14:37:30.718851 2646 ts_tablet_manager.cc:1397] T a625d7d4cd8549e8ad1ecffabbcf5913 P 74f0ef49736744ceaccad487bd25d1cf: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 14:37:30.718855 2647 ts_tablet_manager.cc:1397] T a625d7d4cd8549e8ad1ecffabbcf5913 P ff28141737b8451a813f4524e704bbe7: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260619 14:37:30.721637 2646 raft_consensus.cc:348] T a625d7d4cd8549e8ad1ecffabbcf5913 P 74f0ef49736744ceaccad487bd25d1cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b0bac5d7944fb982708b589db96867" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } } peers { permanent_uuid: "74f0ef49736744ceaccad487bd25d1cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33225 } } peers { permanent_uuid: "ff28141737b8451a813f4524e704bbe7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } }
I20260619 14:37:30.721637 2647 raft_consensus.cc:348] T a625d7d4cd8549e8ad1ecffabbcf5913 P ff28141737b8451a813f4524e704bbe7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b0bac5d7944fb982708b589db96867" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } } peers { permanent_uuid: "74f0ef49736744ceaccad487bd25d1cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33225 } } peers { permanent_uuid: "ff28141737b8451a813f4524e704bbe7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } }
I20260619 14:37:30.721836 2647 raft_consensus.cc:374] T a625d7d4cd8549e8ad1ecffabbcf5913 P ff28141737b8451a813f4524e704bbe7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:30.721840 2646 raft_consensus.cc:374] T a625d7d4cd8549e8ad1ecffabbcf5913 P 74f0ef49736744ceaccad487bd25d1cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:30.721866 2647 raft_consensus.cc:729] T a625d7d4cd8549e8ad1ecffabbcf5913 P ff28141737b8451a813f4524e704bbe7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff28141737b8451a813f4524e704bbe7, State: Initialized, Role: FOLLOWER
I20260619 14:37:30.721886 2646 raft_consensus.cc:729] T a625d7d4cd8549e8ad1ecffabbcf5913 P 74f0ef49736744ceaccad487bd25d1cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74f0ef49736744ceaccad487bd25d1cf, State: Initialized, Role: FOLLOWER
I20260619 14:37:30.722289 2647 consensus_queue.cc:260] T a625d7d4cd8549e8ad1ecffabbcf5913 P ff28141737b8451a813f4524e704bbe7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b0bac5d7944fb982708b589db96867" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } } peers { permanent_uuid: "74f0ef49736744ceaccad487bd25d1cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33225 } } peers { permanent_uuid: "ff28141737b8451a813f4524e704bbe7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } }
I20260619 14:37:30.722313 2646 consensus_queue.cc:260] T a625d7d4cd8549e8ad1ecffabbcf5913 P 74f0ef49736744ceaccad487bd25d1cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b0bac5d7944fb982708b589db96867" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } } peers { permanent_uuid: "74f0ef49736744ceaccad487bd25d1cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33225 } } peers { permanent_uuid: "ff28141737b8451a813f4524e704bbe7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } }
I20260619 14:37:30.723083 2342 heartbeater.cc:503] Master 127.0.98.62:35971 was elected leader, sending a full tablet report...
I20260619 14:37:30.723161 2645 tablet_bootstrap.cc:492] T a625d7d4cd8549e8ad1ecffabbcf5913 P 02b0bac5d7944fb982708b589db96867: No bootstrap required, opened a new log
I20260619 14:37:30.723217 2647 ts_tablet_manager.cc:1428] T a625d7d4cd8549e8ad1ecffabbcf5913 P ff28141737b8451a813f4524e704bbe7: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260619 14:37:30.723230 2646 ts_tablet_manager.cc:1428] T a625d7d4cd8549e8ad1ecffabbcf5913 P 74f0ef49736744ceaccad487bd25d1cf: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260619 14:37:30.723335 2645 ts_tablet_manager.cc:1397] T a625d7d4cd8549e8ad1ecffabbcf5913 P 02b0bac5d7944fb982708b589db96867: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260619 14:37:30.723642 2482 heartbeater.cc:503] Master 127.0.98.62:35971 was elected leader, sending a full tablet report...
I20260619 14:37:30.726752 2645 raft_consensus.cc:348] T a625d7d4cd8549e8ad1ecffabbcf5913 P 02b0bac5d7944fb982708b589db96867 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b0bac5d7944fb982708b589db96867" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } } peers { permanent_uuid: "74f0ef49736744ceaccad487bd25d1cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33225 } } peers { permanent_uuid: "ff28141737b8451a813f4524e704bbe7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } }
I20260619 14:37:30.726949 2645 raft_consensus.cc:374] T a625d7d4cd8549e8ad1ecffabbcf5913 P 02b0bac5d7944fb982708b589db96867 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:30.726982 2645 raft_consensus.cc:729] T a625d7d4cd8549e8ad1ecffabbcf5913 P 02b0bac5d7944fb982708b589db96867 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02b0bac5d7944fb982708b589db96867, State: Initialized, Role: FOLLOWER
I20260619 14:37:30.727367 2645 consensus_queue.cc:260] T a625d7d4cd8549e8ad1ecffabbcf5913 P 02b0bac5d7944fb982708b589db96867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b0bac5d7944fb982708b589db96867" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } } peers { permanent_uuid: "74f0ef49736744ceaccad487bd25d1cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33225 } } peers { permanent_uuid: "ff28141737b8451a813f4524e704bbe7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } }
I20260619 14:37:30.727955 2622 heartbeater.cc:503] Master 127.0.98.62:35971 was elected leader, sending a full tablet report...
I20260619 14:37:30.728228 2645 ts_tablet_manager.cc:1428] T a625d7d4cd8549e8ad1ecffabbcf5913 P 02b0bac5d7944fb982708b589db96867: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260619 14:37:30.736167 2343 tablet.cc:2369] T a625d7d4cd8549e8ad1ecffabbcf5913 P 74f0ef49736744ceaccad487bd25d1cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:37:30.899991 2623 tablet.cc:2369] T a625d7d4cd8549e8ad1ecffabbcf5913 P 02b0bac5d7944fb982708b589db96867: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:37:30.937419 2483 tablet.cc:2369] T a625d7d4cd8549e8ad1ecffabbcf5913 P ff28141737b8451a813f4524e704bbe7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:37:30.952332 2653 raft_consensus.cc:482] T a625d7d4cd8549e8ad1ecffabbcf5913 P 02b0bac5d7944fb982708b589db96867 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:37:30.952524 2653 raft_consensus.cc:504] T a625d7d4cd8549e8ad1ecffabbcf5913 P 02b0bac5d7944fb982708b589db96867 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b0bac5d7944fb982708b589db96867" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } } peers { permanent_uuid: "74f0ef49736744ceaccad487bd25d1cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33225 } } peers { permanent_uuid: "ff28141737b8451a813f4524e704bbe7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } }
I20260619 14:37:30.953899 2653 leader_election.cc:288] T a625d7d4cd8549e8ad1ecffabbcf5913 P 02b0bac5d7944fb982708b589db96867 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74f0ef49736744ceaccad487bd25d1cf (127.0.98.1:33225), ff28141737b8451a813f4524e704bbe7 (127.0.98.2:42505)
I20260619 14:37:30.957825 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:30.954345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52303 (local address 127.0.98.1:33225)
0619 14:37:30.954463 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:30.954468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:30.954481 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:37:30.954538 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:30.954541 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:30.954593 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:30.954674 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:37:30.954963 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:30.955688 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:30.956920 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:30.957328 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:30.957423 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:30.957486 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:30.957649 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 14:37:30.957725 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 14:37:30.957825 2515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:30.954122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33225 (local address 127.0.98.3:52303)
0619 14:37:30.954279 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:30.954295 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:30.954392 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:30.954692 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:30.954695 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:30.954724 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:37:30.954832 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:30.954838 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:30.955794 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:30.955797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:30.956806 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:30.956815 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:30.957442 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:30.957445 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:30.957538 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:30.957555 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 14:37:30.957616 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 14:37:30.957687 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 14:37:30.958465 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a625d7d4cd8549e8ad1ecffabbcf5913" candidate_uuid: "02b0bac5d7944fb982708b589db96867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74f0ef49736744ceaccad487bd25d1cf" is_pre_election: true
I20260619 14:37:30.958788 2291 raft_consensus.cc:2454] T a625d7d4cd8549e8ad1ecffabbcf5913 P 74f0ef49736744ceaccad487bd25d1cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02b0bac5d7944fb982708b589db96867 in term 0.
I20260619 14:37:30.958936 2654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:30.954122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42505 (local address 127.0.98.3:38071)
0619 14:37:30.954734 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:30.954746 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:30.954837 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:30.955044 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:30.955046 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:30.955060 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:37:30.955481 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:30.955487 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:30.956391 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:30.956394 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:30.957607 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:30.957617 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:30.958503 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:30.958507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:30.958595 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:30.958618 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 14:37:30.958733 (+ 115us) client_negotiation.cc:240] Negotiation successful
0619 14:37:30.958798 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"mutex_wait_us":287,"thread_start_us":126,"threads_started":1}
I20260619 14:37:30.958976 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:30.954281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38071 (local address 127.0.98.2:42505)
0619 14:37:30.954394 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:30.954399 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:30.954831 (+ 432us) server_negotiation.cc:407] Connection header received
0619 14:37:30.954916 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:30.954918 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:30.954964 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:30.955027 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:37:30.955595 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:30.956280 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:30.957749 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:30.958368 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:30.958472 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:30.958534 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:30.958772 (+ 238us) server_negotiation.cc:299] Negotiation successful
0619 14:37:30.958857 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 14:37:30.959293 2510 leader_election.cc:302] T a625d7d4cd8549e8ad1ecffabbcf5913 P 02b0bac5d7944fb982708b589db96867 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02b0bac5d7944fb982708b589db96867, 74f0ef49736744ceaccad487bd25d1cf; no voters:
I20260619 14:37:30.959438 2436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a625d7d4cd8549e8ad1ecffabbcf5913" candidate_uuid: "02b0bac5d7944fb982708b589db96867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff28141737b8451a813f4524e704bbe7" is_pre_election: true
I20260619 14:37:30.959563 2653 raft_consensus.cc:2781] T a625d7d4cd8549e8ad1ecffabbcf5913 P 02b0bac5d7944fb982708b589db96867 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:37:30.959648 2653 raft_consensus.cc:482] T a625d7d4cd8549e8ad1ecffabbcf5913 P 02b0bac5d7944fb982708b589db96867 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:37:30.959694 2653 raft_consensus.cc:3037] T a625d7d4cd8549e8ad1ecffabbcf5913 P 02b0bac5d7944fb982708b589db96867 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:30.959730 2436 raft_consensus.cc:2454] T a625d7d4cd8549e8ad1ecffabbcf5913 P ff28141737b8451a813f4524e704bbe7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02b0bac5d7944fb982708b589db96867 in term 0.
I20260619 14:37:30.960927 2653 raft_consensus.cc:504] T a625d7d4cd8549e8ad1ecffabbcf5913 P 02b0bac5d7944fb982708b589db96867 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b0bac5d7944fb982708b589db96867" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } } peers { permanent_uuid: "74f0ef49736744ceaccad487bd25d1cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33225 } } peers { permanent_uuid: "ff28141737b8451a813f4524e704bbe7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } }
I20260619 14:37:30.961263 2653 leader_election.cc:288] T a625d7d4cd8549e8ad1ecffabbcf5913 P 02b0bac5d7944fb982708b589db96867 [CANDIDATE]: Term 1 election: Requested vote from peers 74f0ef49736744ceaccad487bd25d1cf (127.0.98.1:33225), ff28141737b8451a813f4524e704bbe7 (127.0.98.2:42505)
I20260619 14:37:30.961592 2436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a625d7d4cd8549e8ad1ecffabbcf5913" candidate_uuid: "02b0bac5d7944fb982708b589db96867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff28141737b8451a813f4524e704bbe7"
I20260619 14:37:30.961714 2436 raft_consensus.cc:3037] T a625d7d4cd8549e8ad1ecffabbcf5913 P ff28141737b8451a813f4524e704bbe7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:30.962852 2436 raft_consensus.cc:2454] T a625d7d4cd8549e8ad1ecffabbcf5913 P ff28141737b8451a813f4524e704bbe7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02b0bac5d7944fb982708b589db96867 in term 1.
I20260619 14:37:30.963171 2507 leader_election.cc:302] T a625d7d4cd8549e8ad1ecffabbcf5913 P 02b0bac5d7944fb982708b589db96867 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02b0bac5d7944fb982708b589db96867, ff28141737b8451a813f4524e704bbe7; no voters:
I20260619 14:37:30.963155 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a625d7d4cd8549e8ad1ecffabbcf5913" candidate_uuid: "02b0bac5d7944fb982708b589db96867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74f0ef49736744ceaccad487bd25d1cf"
I20260619 14:37:30.963294 2291 raft_consensus.cc:3037] T a625d7d4cd8549e8ad1ecffabbcf5913 P 74f0ef49736744ceaccad487bd25d1cf [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:30.963338 2653 raft_consensus.cc:2781] T a625d7d4cd8549e8ad1ecffabbcf5913 P 02b0bac5d7944fb982708b589db96867 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:37:30.963512 2653 raft_consensus.cc:686] T a625d7d4cd8549e8ad1ecffabbcf5913 P 02b0bac5d7944fb982708b589db96867 [term 1 LEADER]: Becoming Leader. State: Replica: 02b0bac5d7944fb982708b589db96867, State: Running, Role: LEADER
I20260619 14:37:30.963830 2653 consensus_queue.cc:237] T a625d7d4cd8549e8ad1ecffabbcf5913 P 02b0bac5d7944fb982708b589db96867 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b0bac5d7944fb982708b589db96867" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } } peers { permanent_uuid: "74f0ef49736744ceaccad487bd25d1cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33225 } } peers { permanent_uuid: "ff28141737b8451a813f4524e704bbe7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } }
I20260619 14:37:30.964383 2291 raft_consensus.cc:2454] T a625d7d4cd8549e8ad1ecffabbcf5913 P 74f0ef49736744ceaccad487bd25d1cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02b0bac5d7944fb982708b589db96867 in term 1.
I20260619 14:37:30.966815 2137 catalog_manager.cc:5495] T a625d7d4cd8549e8ad1ecffabbcf5913 P 02b0bac5d7944fb982708b589db96867 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02b0bac5d7944fb982708b589db96867 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "02b0bac5d7944fb982708b589db96867" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02b0bac5d7944fb982708b589db96867" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74f0ef49736744ceaccad487bd25d1cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff28141737b8451a813f4524e704bbe7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } health_report { overall_health: UNKNOWN } } }
Jun 19 14:37:31 dist-test-slave-32hj krb5kdc[2094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879849, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:37:31.022002 2662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:31.014432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35971 (local address 127.0.0.1:45258)
0619 14:37:31.014727 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:31.014742 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:31.014843 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:31.015043 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:31.015045 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:31.015427 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:31.015521 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:31.015527 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:31.016184 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:31.016186 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:31.016780 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:31.016788 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:31.017132 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:31.017134 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:31.017194 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:31.017731 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:31.017743 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:31.019107 (+ 1364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:31.020615 (+ 1508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:31.020623 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:31.020628 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:31.020922 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:31.021164 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:31.021167 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:31.021168 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:31.021261 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:31.021548 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:31.021553 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:31.021646 (+ 93us) client_negotiation.cc:769] Sending connection context
0619 14:37:31.021729 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 14:37:31.021845 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":108,"threads_started":1}
I20260619 14:37:31.022017 2188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:31.014460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45258 (local address 127.0.98.62:35971)
0619 14:37:31.014555 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:31.014559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:31.014804 (+ 245us) server_negotiation.cc:407] Connection header received
0619 14:37:31.014906 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:31.014908 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:31.014945 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:31.015017 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:31.015606 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:31.016110 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:31.016859 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:31.017056 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:31.017125 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:31.019233 (+ 2108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:31.019236 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:31.019238 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:31.019258 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:31.020509 (+ 1251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:31.020999 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:31.021002 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:31.021003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:31.021069 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:31.021330 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:31.021332 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:31.021333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:31.021476 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:31.021542 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:31.021760 (+ 218us) server_negotiation.cc:299] Negotiation successful
0619 14:37:31.021896 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":25}
I20260619 14:37:31.025512 2137 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45258:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 14:37:31.026172 2137 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:37:31.030448 2276 tablet_service.cc:1467] Processing CreateTablet for tablet 68a612a4b89349f5bd5077d3cc01121a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=75926fe7eb914163a0ae2f113cca6cd2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 14:37:31.030586 2416 tablet_service.cc:1467] Processing CreateTablet for tablet 68a612a4b89349f5bd5077d3cc01121a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=75926fe7eb914163a0ae2f113cca6cd2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 14:37:31.030586 2556 tablet_service.cc:1467] Processing CreateTablet for tablet 68a612a4b89349f5bd5077d3cc01121a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=75926fe7eb914163a0ae2f113cca6cd2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 14:37:31.031100 2276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68a612a4b89349f5bd5077d3cc01121a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:31.031224 2556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68a612a4b89349f5bd5077d3cc01121a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:31.031219 2416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68a612a4b89349f5bd5077d3cc01121a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:31.033463 2646 tablet_bootstrap.cc:492] T 68a612a4b89349f5bd5077d3cc01121a P 74f0ef49736744ceaccad487bd25d1cf: Bootstrap starting.
I20260619 14:37:31.033949 2645 tablet_bootstrap.cc:492] T 68a612a4b89349f5bd5077d3cc01121a P 02b0bac5d7944fb982708b589db96867: Bootstrap starting.
I20260619 14:37:31.034317 2646 tablet_bootstrap.cc:654] T 68a612a4b89349f5bd5077d3cc01121a P 74f0ef49736744ceaccad487bd25d1cf: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:31.034350 2647 tablet_bootstrap.cc:492] T 68a612a4b89349f5bd5077d3cc01121a P ff28141737b8451a813f4524e704bbe7: Bootstrap starting.
I20260619 14:37:31.034794 2645 tablet_bootstrap.cc:654] T 68a612a4b89349f5bd5077d3cc01121a P 02b0bac5d7944fb982708b589db96867: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:31.035110 2647 tablet_bootstrap.cc:654] T 68a612a4b89349f5bd5077d3cc01121a P ff28141737b8451a813f4524e704bbe7: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:31.035526 2646 tablet_bootstrap.cc:492] T 68a612a4b89349f5bd5077d3cc01121a P 74f0ef49736744ceaccad487bd25d1cf: No bootstrap required, opened a new log
I20260619 14:37:31.035624 2646 ts_tablet_manager.cc:1397] T 68a612a4b89349f5bd5077d3cc01121a P 74f0ef49736744ceaccad487bd25d1cf: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 14:37:31.035876 2645 tablet_bootstrap.cc:492] T 68a612a4b89349f5bd5077d3cc01121a P 02b0bac5d7944fb982708b589db96867: No bootstrap required, opened a new log
I20260619 14:37:31.035956 2645 ts_tablet_manager.cc:1397] T 68a612a4b89349f5bd5077d3cc01121a P 02b0bac5d7944fb982708b589db96867: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 14:37:31.036072 2646 raft_consensus.cc:348] T 68a612a4b89349f5bd5077d3cc01121a P 74f0ef49736744ceaccad487bd25d1cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f0ef49736744ceaccad487bd25d1cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33225 } } peers { permanent_uuid: "ff28141737b8451a813f4524e704bbe7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } } peers { permanent_uuid: "02b0bac5d7944fb982708b589db96867" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } }
I20260619 14:37:31.036192 2646 raft_consensus.cc:374] T 68a612a4b89349f5bd5077d3cc01121a P 74f0ef49736744ceaccad487bd25d1cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:31.036231 2646 raft_consensus.cc:729] T 68a612a4b89349f5bd5077d3cc01121a P 74f0ef49736744ceaccad487bd25d1cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74f0ef49736744ceaccad487bd25d1cf, State: Initialized, Role: FOLLOWER
I20260619 14:37:31.036347 2646 consensus_queue.cc:260] T 68a612a4b89349f5bd5077d3cc01121a P 74f0ef49736744ceaccad487bd25d1cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f0ef49736744ceaccad487bd25d1cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33225 } } peers { permanent_uuid: "ff28141737b8451a813f4524e704bbe7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } } peers { permanent_uuid: "02b0bac5d7944fb982708b589db96867" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } }
I20260619 14:37:31.036372 2645 raft_consensus.cc:348] T 68a612a4b89349f5bd5077d3cc01121a P 02b0bac5d7944fb982708b589db96867 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f0ef49736744ceaccad487bd25d1cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33225 } } peers { permanent_uuid: "ff28141737b8451a813f4524e704bbe7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } } peers { permanent_uuid: "02b0bac5d7944fb982708b589db96867" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } }
I20260619 14:37:31.036456 2645 raft_consensus.cc:374] T 68a612a4b89349f5bd5077d3cc01121a P 02b0bac5d7944fb982708b589db96867 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:31.036491 2645 raft_consensus.cc:729] T 68a612a4b89349f5bd5077d3cc01121a P 02b0bac5d7944fb982708b589db96867 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02b0bac5d7944fb982708b589db96867, State: Initialized, Role: FOLLOWER
I20260619 14:37:31.036623 2646 ts_tablet_manager.cc:1428] T 68a612a4b89349f5bd5077d3cc01121a P 74f0ef49736744ceaccad487bd25d1cf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 14:37:31.036599 2645 consensus_queue.cc:260] T 68a612a4b89349f5bd5077d3cc01121a P 02b0bac5d7944fb982708b589db96867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f0ef49736744ceaccad487bd25d1cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33225 } } peers { permanent_uuid: "ff28141737b8451a813f4524e704bbe7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } } peers { permanent_uuid: "02b0bac5d7944fb982708b589db96867" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } }
I20260619 14:37:31.036615 2652 tablet_replica.cc:439] T 68a612a4b89349f5bd5077d3cc01121a P 74f0ef49736744ceaccad487bd25d1cf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f0ef49736744ceaccad487bd25d1cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33225 } } peers { permanent_uuid: "ff28141737b8451a813f4524e704bbe7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } } peers { permanent_uuid: "02b0bac5d7944fb982708b589db96867" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } } }
I20260619 14:37:31.036832 2652 tablet_replica.cc:442] T 68a612a4b89349f5bd5077d3cc01121a P 74f0ef49736744ceaccad487bd25d1cf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:37:31.036859 2645 ts_tablet_manager.cc:1428] T 68a612a4b89349f5bd5077d3cc01121a P 02b0bac5d7944fb982708b589db96867: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:37:31.036839 2653 tablet_replica.cc:439] T 68a612a4b89349f5bd5077d3cc01121a P 02b0bac5d7944fb982708b589db96867: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f0ef49736744ceaccad487bd25d1cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33225 } } peers { permanent_uuid: "ff28141737b8451a813f4524e704bbe7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } } peers { permanent_uuid: "02b0bac5d7944fb982708b589db96867" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } } }
I20260619 14:37:31.037005 2653 tablet_replica.cc:442] T 68a612a4b89349f5bd5077d3cc01121a P 02b0bac5d7944fb982708b589db96867: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:37:31.037752 2647 tablet_bootstrap.cc:492] T 68a612a4b89349f5bd5077d3cc01121a P ff28141737b8451a813f4524e704bbe7: No bootstrap required, opened a new log
I20260619 14:37:31.037827 2647 ts_tablet_manager.cc:1397] T 68a612a4b89349f5bd5077d3cc01121a P ff28141737b8451a813f4524e704bbe7: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260619 14:37:31.038233 2647 raft_consensus.cc:348] T 68a612a4b89349f5bd5077d3cc01121a P ff28141737b8451a813f4524e704bbe7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f0ef49736744ceaccad487bd25d1cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33225 } } peers { permanent_uuid: "ff28141737b8451a813f4524e704bbe7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } } peers { permanent_uuid: "02b0bac5d7944fb982708b589db96867" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } }
I20260619 14:37:31.038338 2647 raft_consensus.cc:374] T 68a612a4b89349f5bd5077d3cc01121a P ff28141737b8451a813f4524e704bbe7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:31.038376 2647 raft_consensus.cc:729] T 68a612a4b89349f5bd5077d3cc01121a P ff28141737b8451a813f4524e704bbe7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff28141737b8451a813f4524e704bbe7, State: Initialized, Role: FOLLOWER
I20260619 14:37:31.038486 2647 consensus_queue.cc:260] T 68a612a4b89349f5bd5077d3cc01121a P ff28141737b8451a813f4524e704bbe7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f0ef49736744ceaccad487bd25d1cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33225 } } peers { permanent_uuid: "ff28141737b8451a813f4524e704bbe7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } } peers { permanent_uuid: "02b0bac5d7944fb982708b589db96867" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } }
I20260619 14:37:31.038731 2651 tablet_replica.cc:439] T 68a612a4b89349f5bd5077d3cc01121a P ff28141737b8451a813f4524e704bbe7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f0ef49736744ceaccad487bd25d1cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33225 } } peers { permanent_uuid: "ff28141737b8451a813f4524e704bbe7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } } peers { permanent_uuid: "02b0bac5d7944fb982708b589db96867" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } } }
I20260619 14:37:31.038880 2651 tablet_replica.cc:442] T 68a612a4b89349f5bd5077d3cc01121a P ff28141737b8451a813f4524e704bbe7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:37:31.038735 2647 ts_tablet_manager.cc:1428] T 68a612a4b89349f5bd5077d3cc01121a P ff28141737b8451a813f4524e704bbe7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:37:31.175905 2652 raft_consensus.cc:482] T 68a612a4b89349f5bd5077d3cc01121a P 74f0ef49736744ceaccad487bd25d1cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:37:31.176044 2652 raft_consensus.cc:504] T 68a612a4b89349f5bd5077d3cc01121a P 74f0ef49736744ceaccad487bd25d1cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f0ef49736744ceaccad487bd25d1cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33225 } } peers { permanent_uuid: "ff28141737b8451a813f4524e704bbe7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } } peers { permanent_uuid: "02b0bac5d7944fb982708b589db96867" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } }
I20260619 14:37:31.177193 2652 leader_election.cc:288] T 68a612a4b89349f5bd5077d3cc01121a P 74f0ef49736744ceaccad487bd25d1cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff28141737b8451a813f4524e704bbe7 (127.0.98.2:42505), 02b0bac5d7944fb982708b589db96867 (127.0.98.3:33147)
I20260619 14:37:31.180830 2667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:31.177425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42505 (local address 127.0.98.1:57461)
0619 14:37:31.177731 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:31.177742 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:31.177841 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:31.178147 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:31.178149 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:31.178193 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:37:31.178303 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:31.178308 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:31.178996 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:31.179000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:31.179775 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:31.179783 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:31.180393 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:31.180397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:31.180501 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:31.180518 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 14:37:31.180580 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 14:37:31.180643 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":25,"spinlock_wait_cycles":3200,"thread_start_us":75,"threads_started":1}
I20260619 14:37:31.180833 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:31.177486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57461 (local address 127.0.98.2:42505)
0619 14:37:31.177622 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:31.177627 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:31.177817 (+ 190us) server_negotiation.cc:407] Connection header received
0619 14:37:31.177947 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:31.177949 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:31.177988 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:31.178047 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:37:31.178406 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:31.178901 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:31.179890 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:31.180280 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:31.180367 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:31.180429 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:31.180605 (+ 176us) server_negotiation.cc:299] Negotiation successful
0619 14:37:31.180699 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 14:37:31.181016 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:31.177507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49833 (local address 127.0.98.3:33147)
0619 14:37:31.177621 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:31.177626 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:31.177800 (+ 174us) server_negotiation.cc:407] Connection header received
0619 14:37:31.177914 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:31.177917 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:31.177962 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:31.178050 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:37:31.178710 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:31.179181 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:31.180120 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:31.180507 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:31.180594 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:31.180647 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:31.180854 (+ 207us) server_negotiation.cc:299] Negotiation successful
0619 14:37:31.180919 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 14:37:31.181028 2666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:31.177421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33147 (local address 127.0.98.1:49833)
0619 14:37:31.177718 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:31.177728 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:31.177826 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:31.178146 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:31.178148 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:31.178191 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:37:31.178595 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:31.178600 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:31.179287 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:31.179290 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:31.180009 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:31.180017 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:31.180609 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:31.180613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:31.180708 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:31.180751 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 14:37:31.180815 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 14:37:31.180881 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":25,"spinlock_wait_cycles":40448,"thread_start_us":76,"threads_started":1}
I20260619 14:37:31.181300 2436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a612a4b89349f5bd5077d3cc01121a" candidate_uuid: "74f0ef49736744ceaccad487bd25d1cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff28141737b8451a813f4524e704bbe7" is_pre_election: true
I20260619 14:37:31.181452 2436 raft_consensus.cc:2454] T 68a612a4b89349f5bd5077d3cc01121a P ff28141737b8451a813f4524e704bbe7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74f0ef49736744ceaccad487bd25d1cf in term 0.
I20260619 14:37:31.181680 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a612a4b89349f5bd5077d3cc01121a" candidate_uuid: "74f0ef49736744ceaccad487bd25d1cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02b0bac5d7944fb982708b589db96867" is_pre_election: true
I20260619 14:37:31.181804 2227 leader_election.cc:302] T 68a612a4b89349f5bd5077d3cc01121a P 74f0ef49736744ceaccad487bd25d1cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74f0ef49736744ceaccad487bd25d1cf, ff28141737b8451a813f4524e704bbe7; no voters:
I20260619 14:37:31.181895 2576 raft_consensus.cc:2454] T 68a612a4b89349f5bd5077d3cc01121a P 02b0bac5d7944fb982708b589db96867 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74f0ef49736744ceaccad487bd25d1cf in term 0.
I20260619 14:37:31.182091 2652 raft_consensus.cc:2781] T 68a612a4b89349f5bd5077d3cc01121a P 74f0ef49736744ceaccad487bd25d1cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:37:31.182155 2652 raft_consensus.cc:482] T 68a612a4b89349f5bd5077d3cc01121a P 74f0ef49736744ceaccad487bd25d1cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:37:31.182193 2652 raft_consensus.cc:3037] T 68a612a4b89349f5bd5077d3cc01121a P 74f0ef49736744ceaccad487bd25d1cf [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:31.183109 2652 raft_consensus.cc:504] T 68a612a4b89349f5bd5077d3cc01121a P 74f0ef49736744ceaccad487bd25d1cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f0ef49736744ceaccad487bd25d1cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33225 } } peers { permanent_uuid: "ff28141737b8451a813f4524e704bbe7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } } peers { permanent_uuid: "02b0bac5d7944fb982708b589db96867" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } }
I20260619 14:37:31.183413 2652 leader_election.cc:288] T 68a612a4b89349f5bd5077d3cc01121a P 74f0ef49736744ceaccad487bd25d1cf [CANDIDATE]: Term 1 election: Requested vote from peers ff28141737b8451a813f4524e704bbe7 (127.0.98.2:42505), 02b0bac5d7944fb982708b589db96867 (127.0.98.3:33147)
I20260619 14:37:31.183723 2436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a612a4b89349f5bd5077d3cc01121a" candidate_uuid: "74f0ef49736744ceaccad487bd25d1cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff28141737b8451a813f4524e704bbe7"
I20260619 14:37:31.183748 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a612a4b89349f5bd5077d3cc01121a" candidate_uuid: "74f0ef49736744ceaccad487bd25d1cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02b0bac5d7944fb982708b589db96867"
I20260619 14:37:31.183840 2576 raft_consensus.cc:3037] T 68a612a4b89349f5bd5077d3cc01121a P 02b0bac5d7944fb982708b589db96867 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:31.183840 2436 raft_consensus.cc:3037] T 68a612a4b89349f5bd5077d3cc01121a P ff28141737b8451a813f4524e704bbe7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:31.184876 2576 raft_consensus.cc:2454] T 68a612a4b89349f5bd5077d3cc01121a P 02b0bac5d7944fb982708b589db96867 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74f0ef49736744ceaccad487bd25d1cf in term 1.
I20260619 14:37:31.184876 2436 raft_consensus.cc:2454] T 68a612a4b89349f5bd5077d3cc01121a P ff28141737b8451a813f4524e704bbe7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74f0ef49736744ceaccad487bd25d1cf in term 1.
I20260619 14:37:31.185134 2229 leader_election.cc:302] T 68a612a4b89349f5bd5077d3cc01121a P 74f0ef49736744ceaccad487bd25d1cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02b0bac5d7944fb982708b589db96867, 74f0ef49736744ceaccad487bd25d1cf; no voters:
I20260619 14:37:31.185339 2652 raft_consensus.cc:2781] T 68a612a4b89349f5bd5077d3cc01121a P 74f0ef49736744ceaccad487bd25d1cf [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:37:31.185523 2652 raft_consensus.cc:686] T 68a612a4b89349f5bd5077d3cc01121a P 74f0ef49736744ceaccad487bd25d1cf [term 1 LEADER]: Becoming Leader. State: Replica: 74f0ef49736744ceaccad487bd25d1cf, State: Running, Role: LEADER
I20260619 14:37:31.185842 2652 consensus_queue.cc:237] T 68a612a4b89349f5bd5077d3cc01121a P 74f0ef49736744ceaccad487bd25d1cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f0ef49736744ceaccad487bd25d1cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33225 } } peers { permanent_uuid: "ff28141737b8451a813f4524e704bbe7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } } peers { permanent_uuid: "02b0bac5d7944fb982708b589db96867" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } }
I20260619 14:37:31.187985 2668 tablet_replica.cc:439] T 68a612a4b89349f5bd5077d3cc01121a P 74f0ef49736744ceaccad487bd25d1cf: TxnStatusTablet state changed. Reason: New leader 74f0ef49736744ceaccad487bd25d1cf. Latest consensus state: current_term: 1 leader_uuid: "74f0ef49736744ceaccad487bd25d1cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f0ef49736744ceaccad487bd25d1cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33225 } } peers { permanent_uuid: "ff28141737b8451a813f4524e704bbe7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } } peers { permanent_uuid: "02b0bac5d7944fb982708b589db96867" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } } }
I20260619 14:37:31.188108 2668 tablet_replica.cc:442] T 68a612a4b89349f5bd5077d3cc01121a P 74f0ef49736744ceaccad487bd25d1cf: This TxnStatusTablet replica's current role is: LEADER
I20260619 14:37:31.188366 2670 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 14:37:31.188553 2670 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 14:37:31.188985 2138 catalog_manager.cc:5495] T 68a612a4b89349f5bd5077d3cc01121a P 74f0ef49736744ceaccad487bd25d1cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 74f0ef49736744ceaccad487bd25d1cf (127.0.98.1). New cstate: current_term: 1 leader_uuid: "74f0ef49736744ceaccad487bd25d1cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f0ef49736744ceaccad487bd25d1cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff28141737b8451a813f4524e704bbe7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02b0bac5d7944fb982708b589db96867" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } health_report { overall_health: UNKNOWN } } }
I20260619 14:37:31.231942 2662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:31.228092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33225 (local address 127.0.0.1:53818)
0619 14:37:31.228225 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:31.228239 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:31.228378 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:31.228683 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:31.228685 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:31.228692 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:37:31.228800 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:31.228804 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:31.229769 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:31.229771 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:31.230367 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:31.230375 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:31.230893 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:31.230895 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:31.230952 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:31.230990 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:37:31.231696 (+ 706us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:37:31.231710 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 14:37:31.231765 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 14:37:31.231837 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260619 14:37:31.231988 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:31.228182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53818 (local address 127.0.98.1:33225)
0619 14:37:31.228301 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:31.228305 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:31.228320 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:37:31.228511 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:31.228514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:31.228579 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:31.228682 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:37:31.228888 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:31.229671 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:31.230464 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:31.230802 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:31.230900 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:31.231100 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:37:31.231602 (+ 502us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:37:31.231710 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:31.231777 (+ 67us) server_negotiation.cc:299] Negotiation successful
0619 14:37:31.231829 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 14:37:31.237080 2576 raft_consensus.cc:1261] T 68a612a4b89349f5bd5077d3cc01121a P 02b0bac5d7944fb982708b589db96867 [term 1 FOLLOWER]: Refusing update from remote peer 74f0ef49736744ceaccad487bd25d1cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:37:31.237159 2436 raft_consensus.cc:1261] T 68a612a4b89349f5bd5077d3cc01121a P ff28141737b8451a813f4524e704bbe7 [term 1 FOLLOWER]: Refusing update from remote peer 74f0ef49736744ceaccad487bd25d1cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:37:31.237844 2652 consensus_queue.cc:1035] T 68a612a4b89349f5bd5077d3cc01121a P 74f0ef49736744ceaccad487bd25d1cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff28141737b8451a813f4524e704bbe7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:37:31.238080 2668 consensus_queue.cc:1035] T 68a612a4b89349f5bd5077d3cc01121a P 74f0ef49736744ceaccad487bd25d1cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "02b0bac5d7944fb982708b589db96867" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:37:31.241072 2653 tablet_replica.cc:439] T 68a612a4b89349f5bd5077d3cc01121a P 02b0bac5d7944fb982708b589db96867: TxnStatusTablet state changed. Reason: New leader 74f0ef49736744ceaccad487bd25d1cf. Latest consensus state: current_term: 1 leader_uuid: "74f0ef49736744ceaccad487bd25d1cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f0ef49736744ceaccad487bd25d1cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33225 } } peers { permanent_uuid: "ff28141737b8451a813f4524e704bbe7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } } peers { permanent_uuid: "02b0bac5d7944fb982708b589db96867" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } } }
I20260619 14:37:31.241180 2653 tablet_replica.cc:442] T 68a612a4b89349f5bd5077d3cc01121a P 02b0bac5d7944fb982708b589db96867: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:37:31.241554 2651 tablet_replica.cc:439] T 68a612a4b89349f5bd5077d3cc01121a P ff28141737b8451a813f4524e704bbe7: TxnStatusTablet state changed. Reason: New leader 74f0ef49736744ceaccad487bd25d1cf. Latest consensus state: current_term: 1 leader_uuid: "74f0ef49736744ceaccad487bd25d1cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f0ef49736744ceaccad487bd25d1cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33225 } } peers { permanent_uuid: "ff28141737b8451a813f4524e704bbe7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } } peers { permanent_uuid: "02b0bac5d7944fb982708b589db96867" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } } }
I20260619 14:37:31.241652 2651 tablet_replica.cc:442] T 68a612a4b89349f5bd5077d3cc01121a P ff28141737b8451a813f4524e704bbe7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:37:31.244611 2652 tablet_replica.cc:439] T 68a612a4b89349f5bd5077d3cc01121a P 74f0ef49736744ceaccad487bd25d1cf: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "74f0ef49736744ceaccad487bd25d1cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f0ef49736744ceaccad487bd25d1cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33225 } } peers { permanent_uuid: "ff28141737b8451a813f4524e704bbe7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } } peers { permanent_uuid: "02b0bac5d7944fb982708b589db96867" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } } }
I20260619 14:37:31.244750 2652 tablet_replica.cc:442] T 68a612a4b89349f5bd5077d3cc01121a P 74f0ef49736744ceaccad487bd25d1cf: This TxnStatusTablet replica's current role is: LEADER
I20260619 14:37:31.245347 2652 tablet_replica.cc:439] T 68a612a4b89349f5bd5077d3cc01121a P 74f0ef49736744ceaccad487bd25d1cf: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "74f0ef49736744ceaccad487bd25d1cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f0ef49736744ceaccad487bd25d1cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33225 } } peers { permanent_uuid: "ff28141737b8451a813f4524e704bbe7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } } peers { permanent_uuid: "02b0bac5d7944fb982708b589db96867" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } } }
I20260619 14:37:31.245442 2652 tablet_replica.cc:442] T 68a612a4b89349f5bd5077d3cc01121a P 74f0ef49736744ceaccad487bd25d1cf: This TxnStatusTablet replica's current role is: LEADER
I20260619 14:37:31.246419 2651 tablet_replica.cc:439] T 68a612a4b89349f5bd5077d3cc01121a P ff28141737b8451a813f4524e704bbe7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "74f0ef49736744ceaccad487bd25d1cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f0ef49736744ceaccad487bd25d1cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33225 } } peers { permanent_uuid: "ff28141737b8451a813f4524e704bbe7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } } peers { permanent_uuid: "02b0bac5d7944fb982708b589db96867" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } } }
I20260619 14:37:31.246515 2651 tablet_replica.cc:442] T 68a612a4b89349f5bd5077d3cc01121a P ff28141737b8451a813f4524e704bbe7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:37:31.248116 2675 mvcc.cc:205] Tried to move back new op lower bound from 7298579870660579328 to 7298579870459842560. Current Snapshot: MvccSnapshot[applied={T|T < 7298579870660579328}]
I20260619 14:37:31.248324 2653 tablet_replica.cc:439] T 68a612a4b89349f5bd5077d3cc01121a P 02b0bac5d7944fb982708b589db96867: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "74f0ef49736744ceaccad487bd25d1cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f0ef49736744ceaccad487bd25d1cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33225 } } peers { permanent_uuid: "ff28141737b8451a813f4524e704bbe7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } } peers { permanent_uuid: "02b0bac5d7944fb982708b589db96867" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } } }
I20260619 14:37:31.248433 2653 tablet_replica.cc:442] T 68a612a4b89349f5bd5077d3cc01121a P 02b0bac5d7944fb982708b589db96867: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:37:31.250155 2674 mvcc.cc:205] Tried to move back new op lower bound from 7298579870660579328 to 7298579870459842560. Current Snapshot: MvccSnapshot[applied={T|T < 7298579870660579328}]
I20260619 14:37:31.257045 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:31.253832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45348 (local address 127.0.98.3:33147)
0619 14:37:31.253955 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:31.253959 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:31.253972 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:37:31.254042 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:31.254045 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:31.254095 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:31.254181 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:37:31.254412 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:31.254924 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:31.255748 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:31.255958 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:31.256045 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:31.256291 (+ 246us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:37:31.256672 (+ 381us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:37:31.256797 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:31.256869 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 14:37:31.256920 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 14:37:31.267233 2686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:31.263787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33225 (local address 127.0.98.3:42651)
0619 14:37:31.264046 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:31.264058 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:31.264139 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:31.264403 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:31.264406 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:31.264413 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:37:31.264512 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:31.264516 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:31.265514 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:31.265517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:31.266118 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:31.266126 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:31.266633 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:31.266635 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:31.266700 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:31.266737 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:37:31.267002 (+ 265us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:37:31.267005 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 14:37:31.267043 (+ 38us) client_negotiation.cc:240] Negotiation successful
0619 14:37:31.267097 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":89,"threads_started":1}
I20260619 14:37:31.267232 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:31.263888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42651 (local address 127.0.98.1:33225)
0619 14:37:31.264078 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:31.264084 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:31.264101 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:37:31.264230 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:31.264233 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:31.264282 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:31.264355 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:37:31.264606 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:31.265399 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:31.266220 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:31.266544 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:31.266641 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:31.266861 (+ 220us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:37:31.266934 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:37:31.266991 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:31.267058 (+ 67us) server_negotiation.cc:299] Negotiation successful
0619 14:37:31.267110 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260619 14:37:31.273411 2436 raft_consensus.cc:1261] T a625d7d4cd8549e8ad1ecffabbcf5913 P ff28141737b8451a813f4524e704bbe7 [term 1 FOLLOWER]: Refusing update from remote peer 02b0bac5d7944fb982708b589db96867: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:37:31.273801 2653 consensus_queue.cc:1035] T a625d7d4cd8549e8ad1ecffabbcf5913 P 02b0bac5d7944fb982708b589db96867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff28141737b8451a813f4524e704bbe7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:37:31.273912 2291 raft_consensus.cc:1261] T a625d7d4cd8549e8ad1ecffabbcf5913 P 74f0ef49736744ceaccad487bd25d1cf [term 1 FOLLOWER]: Refusing update from remote peer 02b0bac5d7944fb982708b589db96867: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:37:31.274308 2653 consensus_queue.cc:1035] T a625d7d4cd8549e8ad1ecffabbcf5913 P 02b0bac5d7944fb982708b589db96867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74f0ef49736744ceaccad487bd25d1cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:37:31.282526 2673 mvcc.cc:205] Tried to move back new op lower bound from 7298579870811820032 to 7298579869550362624. Current Snapshot: MvccSnapshot[applied={T|T < 7298579870811820032}]
I20260619 14:37:31.294745 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:31.290969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36127 (local address 127.0.98.3:33147)
0619 14:37:31.291080 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:31.291084 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:31.291316 (+ 232us) server_negotiation.cc:407] Connection header received
0619 14:37:31.291415 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:31.291418 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:31.291462 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:31.291519 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:37:31.291760 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:31.292516 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:31.293800 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:31.294006 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:31.294083 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:31.294386 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:37:31.294445 (+ 59us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:37:31.294499 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:31.294590 (+ 91us) server_negotiation.cc:299] Negotiation successful
0619 14:37:31.294639 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 14:37:31.295006 2692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:31.290884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33147 (local address 127.0.98.1:36127)
0619 14:37:31.291239 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:31.291250 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:31.291341 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:31.291536 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:31.291539 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:31.291546 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:37:31.291650 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:31.291658 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:31.292657 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:31.292661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:31.293676 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:31.293689 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:31.294129 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:31.294133 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:31.294227 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:31.294276 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:37:31.294516 (+ 240us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:37:31.294521 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 14:37:31.294580 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 14:37:31.294655 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":87,"threads_started":1}
I20260619 14:37:31.320765 2138 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:37:31.320964 2138 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:37:31.323534 2138 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 96890e15464d40b1a3c6643bcb0abd9a: Sending DeleteTablet for 3 replicas of tablet a625d7d4cd8549e8ad1ecffabbcf5913
I20260619 14:37:31.324324 2556 tablet_service.cc:1514] Processing DeleteTablet for tablet a625d7d4cd8549e8ad1ecffabbcf5913 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:37:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45340
I20260619 14:37:31.324448 2416 tablet_service.cc:1514] Processing DeleteTablet for tablet a625d7d4cd8549e8ad1ecffabbcf5913 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:37:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33156
I20260619 14:37:31.324932 2276 tablet_service.cc:1514] Processing DeleteTablet for tablet a625d7d4cd8549e8ad1ecffabbcf5913 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:37:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53802
I20260619 14:37:31.325146 2695 tablet_replica.cc:330] T a625d7d4cd8549e8ad1ecffabbcf5913 P 02b0bac5d7944fb982708b589db96867: stopping tablet replica
I20260619 14:37:31.325481 2695 raft_consensus.cc:2229] T a625d7d4cd8549e8ad1ecffabbcf5913 P 02b0bac5d7944fb982708b589db96867 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:37:31.325891 2695 raft_consensus.cc:2258] T a625d7d4cd8549e8ad1ecffabbcf5913 P 02b0bac5d7944fb982708b589db96867 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:37:31.325932 2138 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45248:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:37:31.326349 2138 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:37:31.327545 2695 ts_tablet_manager.cc:1905] T a625d7d4cd8549e8ad1ecffabbcf5913 P 02b0bac5d7944fb982708b589db96867: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:37:31.327943 2697 tablet_replica.cc:330] T a625d7d4cd8549e8ad1ecffabbcf5913 P 74f0ef49736744ceaccad487bd25d1cf: stopping tablet replica
I20260619 14:37:31.328287 2697 raft_consensus.cc:2229] T a625d7d4cd8549e8ad1ecffabbcf5913 P 74f0ef49736744ceaccad487bd25d1cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:37:31.328431 2697 raft_consensus.cc:2258] T a625d7d4cd8549e8ad1ecffabbcf5913 P 74f0ef49736744ceaccad487bd25d1cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:37:31.328745 2696 tablet_replica.cc:330] T a625d7d4cd8549e8ad1ecffabbcf5913 P ff28141737b8451a813f4524e704bbe7: stopping tablet replica
I20260619 14:37:31.329010 2696 raft_consensus.cc:2229] T a625d7d4cd8549e8ad1ecffabbcf5913 P ff28141737b8451a813f4524e704bbe7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:37:31.332278 2276 tablet_service.cc:1467] Processing CreateTablet for tablet db2e0c4eb757451691f5873fe707d4e1 (DEFAULT_TABLE table=test-table [id=5ba2d5d51ad5488fa8a65d663d303457]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:37:31.332546 2276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db2e0c4eb757451691f5873fe707d4e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:31.332809 2695 ts_tablet_manager.cc:1918] T a625d7d4cd8549e8ad1ecffabbcf5913 P 02b0bac5d7944fb982708b589db96867: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:37:31.332963 2695 log.cc:1196] T a625d7d4cd8549e8ad1ecffabbcf5913 P 02b0bac5d7944fb982708b589db96867: Deleting WAL directory at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/ts-2/wal/wals/a625d7d4cd8549e8ad1ecffabbcf5913
I20260619 14:37:31.333312 2695 ts_tablet_manager.cc:1939] T a625d7d4cd8549e8ad1ecffabbcf5913 P 02b0bac5d7944fb982708b589db96867: Deleting consensus metadata
I20260619 14:37:31.334915 2646 tablet_bootstrap.cc:492] T db2e0c4eb757451691f5873fe707d4e1 P 74f0ef49736744ceaccad487bd25d1cf: Bootstrap starting.
I20260619 14:37:31.335557 2125 catalog_manager.cc:4841] TS 02b0bac5d7944fb982708b589db96867 (127.0.98.3:33147): tablet a625d7d4cd8549e8ad1ecffabbcf5913 (table test-table [id=b4c39e646d184870a0c64ac613f9149e]) successfully deleted
I20260619 14:37:31.335995 2556 tablet_service.cc:1467] Processing CreateTablet for tablet db2e0c4eb757451691f5873fe707d4e1 (DEFAULT_TABLE table=test-table [id=5ba2d5d51ad5488fa8a65d663d303457]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:37:31.336239 2556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db2e0c4eb757451691f5873fe707d4e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:31.336258 2646 tablet_bootstrap.cc:654] T db2e0c4eb757451691f5873fe707d4e1 P 74f0ef49736744ceaccad487bd25d1cf: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:31.337422 2646 tablet_bootstrap.cc:492] T db2e0c4eb757451691f5873fe707d4e1 P 74f0ef49736744ceaccad487bd25d1cf: No bootstrap required, opened a new log
I20260619 14:37:31.337497 2646 ts_tablet_manager.cc:1397] T db2e0c4eb757451691f5873fe707d4e1 P 74f0ef49736744ceaccad487bd25d1cf: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 14:37:31.337877 2646 raft_consensus.cc:348] T db2e0c4eb757451691f5873fe707d4e1 P 74f0ef49736744ceaccad487bd25d1cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f0ef49736744ceaccad487bd25d1cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33225 } } peers { permanent_uuid: "ff28141737b8451a813f4524e704bbe7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } } peers { permanent_uuid: "02b0bac5d7944fb982708b589db96867" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } }
I20260619 14:37:31.337973 2646 raft_consensus.cc:374] T db2e0c4eb757451691f5873fe707d4e1 P 74f0ef49736744ceaccad487bd25d1cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:31.338001 2646 raft_consensus.cc:729] T db2e0c4eb757451691f5873fe707d4e1 P 74f0ef49736744ceaccad487bd25d1cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74f0ef49736744ceaccad487bd25d1cf, State: Initialized, Role: FOLLOWER
I20260619 14:37:31.338110 2646 consensus_queue.cc:260] T db2e0c4eb757451691f5873fe707d4e1 P 74f0ef49736744ceaccad487bd25d1cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f0ef49736744ceaccad487bd25d1cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33225 } } peers { permanent_uuid: "ff28141737b8451a813f4524e704bbe7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } } peers { permanent_uuid: "02b0bac5d7944fb982708b589db96867" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } }
I20260619 14:37:31.338263 2645 tablet_bootstrap.cc:492] T db2e0c4eb757451691f5873fe707d4e1 P 02b0bac5d7944fb982708b589db96867: Bootstrap starting.
I20260619 14:37:31.338359 2646 ts_tablet_manager.cc:1428] T db2e0c4eb757451691f5873fe707d4e1 P 74f0ef49736744ceaccad487bd25d1cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:37:31.339121 2645 tablet_bootstrap.cc:654] T db2e0c4eb757451691f5873fe707d4e1 P 02b0bac5d7944fb982708b589db96867: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:31.340131 2697 ts_tablet_manager.cc:1905] T a625d7d4cd8549e8ad1ecffabbcf5913 P 74f0ef49736744ceaccad487bd25d1cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:37:31.329195 2696 raft_consensus.cc:2258] T a625d7d4cd8549e8ad1ecffabbcf5913 P ff28141737b8451a813f4524e704bbe7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:37:31.335089 2416 tablet_service.cc:1467] Processing CreateTablet for tablet db2e0c4eb757451691f5873fe707d4e1 (DEFAULT_TABLE table=test-table [id=5ba2d5d51ad5488fa8a65d663d303457]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:37:31.341720 2416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db2e0c4eb757451691f5873fe707d4e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:31.344026 2647 tablet_bootstrap.cc:492] T db2e0c4eb757451691f5873fe707d4e1 P ff28141737b8451a813f4524e704bbe7: Bootstrap starting.
I20260619 14:37:31.344933 2647 tablet_bootstrap.cc:654] T db2e0c4eb757451691f5873fe707d4e1 P ff28141737b8451a813f4524e704bbe7: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:31.345125 2696 ts_tablet_manager.cc:1905] T a625d7d4cd8549e8ad1ecffabbcf5913 P ff28141737b8451a813f4524e704bbe7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:37:31.346285 2647 tablet_bootstrap.cc:492] T db2e0c4eb757451691f5873fe707d4e1 P ff28141737b8451a813f4524e704bbe7: No bootstrap required, opened a new log
I20260619 14:37:31.346359 2647 ts_tablet_manager.cc:1397] T db2e0c4eb757451691f5873fe707d4e1 P ff28141737b8451a813f4524e704bbe7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 14:37:31.346742 2647 raft_consensus.cc:348] T db2e0c4eb757451691f5873fe707d4e1 P ff28141737b8451a813f4524e704bbe7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f0ef49736744ceaccad487bd25d1cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33225 } } peers { permanent_uuid: "ff28141737b8451a813f4524e704bbe7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } } peers { permanent_uuid: "02b0bac5d7944fb982708b589db96867" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } }
I20260619 14:37:31.346838 2647 raft_consensus.cc:374] T db2e0c4eb757451691f5873fe707d4e1 P ff28141737b8451a813f4524e704bbe7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:31.346866 2647 raft_consensus.cc:729] T db2e0c4eb757451691f5873fe707d4e1 P ff28141737b8451a813f4524e704bbe7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff28141737b8451a813f4524e704bbe7, State: Initialized, Role: FOLLOWER
I20260619 14:37:31.346972 2647 consensus_queue.cc:260] T db2e0c4eb757451691f5873fe707d4e1 P ff28141737b8451a813f4524e704bbe7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f0ef49736744ceaccad487bd25d1cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33225 } } peers { permanent_uuid: "ff28141737b8451a813f4524e704bbe7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } } peers { permanent_uuid: "02b0bac5d7944fb982708b589db96867" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } }
I20260619 14:37:31.347200 2647 ts_tablet_manager.cc:1428] T db2e0c4eb757451691f5873fe707d4e1 P ff28141737b8451a813f4524e704bbe7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:37:31.347697 2697 ts_tablet_manager.cc:1918] T a625d7d4cd8549e8ad1ecffabbcf5913 P 74f0ef49736744ceaccad487bd25d1cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:37:31.347779 2697 log.cc:1196] T a625d7d4cd8549e8ad1ecffabbcf5913 P 74f0ef49736744ceaccad487bd25d1cf: Deleting WAL directory at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/ts-0/wal/wals/a625d7d4cd8549e8ad1ecffabbcf5913
I20260619 14:37:31.348030 2697 ts_tablet_manager.cc:1939] T a625d7d4cd8549e8ad1ecffabbcf5913 P 74f0ef49736744ceaccad487bd25d1cf: Deleting consensus metadata
I20260619 14:37:31.349857 2696 ts_tablet_manager.cc:1918] T a625d7d4cd8549e8ad1ecffabbcf5913 P ff28141737b8451a813f4524e704bbe7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:37:31.349942 2696 log.cc:1196] T a625d7d4cd8549e8ad1ecffabbcf5913 P ff28141737b8451a813f4524e704bbe7: Deleting WAL directory at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/ts-1/wal/wals/a625d7d4cd8549e8ad1ecffabbcf5913
I20260619 14:37:31.350205 2696 ts_tablet_manager.cc:1939] T a625d7d4cd8549e8ad1ecffabbcf5913 P ff28141737b8451a813f4524e704bbe7: Deleting consensus metadata
I20260619 14:37:31.350737 2126 catalog_manager.cc:4841] TS 74f0ef49736744ceaccad487bd25d1cf (127.0.98.1:33225): tablet a625d7d4cd8549e8ad1ecffabbcf5913 (table test-table [id=b4c39e646d184870a0c64ac613f9149e]) successfully deleted
I20260619 14:37:31.352130 2123 catalog_manager.cc:4841] TS ff28141737b8451a813f4524e704bbe7 (127.0.98.2:42505): tablet a625d7d4cd8549e8ad1ecffabbcf5913 (table test-table [id=b4c39e646d184870a0c64ac613f9149e]) successfully deleted
I20260619 14:37:31.352860 2645 tablet_bootstrap.cc:492] T db2e0c4eb757451691f5873fe707d4e1 P 02b0bac5d7944fb982708b589db96867: No bootstrap required, opened a new log
I20260619 14:37:31.352960 2645 ts_tablet_manager.cc:1397] T db2e0c4eb757451691f5873fe707d4e1 P 02b0bac5d7944fb982708b589db96867: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20260619 14:37:31.353420 2645 raft_consensus.cc:348] T db2e0c4eb757451691f5873fe707d4e1 P 02b0bac5d7944fb982708b589db96867 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f0ef49736744ceaccad487bd25d1cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33225 } } peers { permanent_uuid: "ff28141737b8451a813f4524e704bbe7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } } peers { permanent_uuid: "02b0bac5d7944fb982708b589db96867" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } }
I20260619 14:37:31.353538 2645 raft_consensus.cc:374] T db2e0c4eb757451691f5873fe707d4e1 P 02b0bac5d7944fb982708b589db96867 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:31.353574 2645 raft_consensus.cc:729] T db2e0c4eb757451691f5873fe707d4e1 P 02b0bac5d7944fb982708b589db96867 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02b0bac5d7944fb982708b589db96867, State: Initialized, Role: FOLLOWER
I20260619 14:37:31.353693 2645 consensus_queue.cc:260] T db2e0c4eb757451691f5873fe707d4e1 P 02b0bac5d7944fb982708b589db96867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f0ef49736744ceaccad487bd25d1cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33225 } } peers { permanent_uuid: "ff28141737b8451a813f4524e704bbe7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } } peers { permanent_uuid: "02b0bac5d7944fb982708b589db96867" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } }
I20260619 14:37:31.353953 2645 ts_tablet_manager.cc:1428] T db2e0c4eb757451691f5873fe707d4e1 P 02b0bac5d7944fb982708b589db96867: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:37:31.550282 2655 raft_consensus.cc:482] T db2e0c4eb757451691f5873fe707d4e1 P 02b0bac5d7944fb982708b589db96867 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:37:31.550463 2655 raft_consensus.cc:504] T db2e0c4eb757451691f5873fe707d4e1 P 02b0bac5d7944fb982708b589db96867 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f0ef49736744ceaccad487bd25d1cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33225 } } peers { permanent_uuid: "ff28141737b8451a813f4524e704bbe7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } } peers { permanent_uuid: "02b0bac5d7944fb982708b589db96867" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } }
I20260619 14:37:31.550907 2655 leader_election.cc:288] T db2e0c4eb757451691f5873fe707d4e1 P 02b0bac5d7944fb982708b589db96867 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74f0ef49736744ceaccad487bd25d1cf (127.0.98.1:33225), ff28141737b8451a813f4524e704bbe7 (127.0.98.2:42505)
I20260619 14:37:31.551375 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db2e0c4eb757451691f5873fe707d4e1" candidate_uuid: "02b0bac5d7944fb982708b589db96867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74f0ef49736744ceaccad487bd25d1cf" is_pre_election: true
I20260619 14:37:31.551375 2436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db2e0c4eb757451691f5873fe707d4e1" candidate_uuid: "02b0bac5d7944fb982708b589db96867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff28141737b8451a813f4524e704bbe7" is_pre_election: true
I20260619 14:37:31.551529 2291 raft_consensus.cc:2454] T db2e0c4eb757451691f5873fe707d4e1 P 74f0ef49736744ceaccad487bd25d1cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02b0bac5d7944fb982708b589db96867 in term 0.
I20260619 14:37:31.551529 2436 raft_consensus.cc:2454] T db2e0c4eb757451691f5873fe707d4e1 P ff28141737b8451a813f4524e704bbe7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02b0bac5d7944fb982708b589db96867 in term 0.
I20260619 14:37:31.551832 2510 leader_election.cc:302] T db2e0c4eb757451691f5873fe707d4e1 P 02b0bac5d7944fb982708b589db96867 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 02b0bac5d7944fb982708b589db96867, 74f0ef49736744ceaccad487bd25d1cf, ff28141737b8451a813f4524e704bbe7; no voters:
I20260619 14:37:31.552008 2655 raft_consensus.cc:2781] T db2e0c4eb757451691f5873fe707d4e1 P 02b0bac5d7944fb982708b589db96867 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:37:31.552070 2655 raft_consensus.cc:482] T db2e0c4eb757451691f5873fe707d4e1 P 02b0bac5d7944fb982708b589db96867 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:37:31.552107 2655 raft_consensus.cc:3037] T db2e0c4eb757451691f5873fe707d4e1 P 02b0bac5d7944fb982708b589db96867 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:31.553126 2655 raft_consensus.cc:504] T db2e0c4eb757451691f5873fe707d4e1 P 02b0bac5d7944fb982708b589db96867 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f0ef49736744ceaccad487bd25d1cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33225 } } peers { permanent_uuid: "ff28141737b8451a813f4524e704bbe7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } } peers { permanent_uuid: "02b0bac5d7944fb982708b589db96867" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } }
I20260619 14:37:31.553422 2655 leader_election.cc:288] T db2e0c4eb757451691f5873fe707d4e1 P 02b0bac5d7944fb982708b589db96867 [CANDIDATE]: Term 1 election: Requested vote from peers 74f0ef49736744ceaccad487bd25d1cf (127.0.98.1:33225), ff28141737b8451a813f4524e704bbe7 (127.0.98.2:42505)
I20260619 14:37:31.553738 2436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db2e0c4eb757451691f5873fe707d4e1" candidate_uuid: "02b0bac5d7944fb982708b589db96867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff28141737b8451a813f4524e704bbe7"
I20260619 14:37:31.553774 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db2e0c4eb757451691f5873fe707d4e1" candidate_uuid: "02b0bac5d7944fb982708b589db96867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74f0ef49736744ceaccad487bd25d1cf"
I20260619 14:37:31.553838 2436 raft_consensus.cc:3037] T db2e0c4eb757451691f5873fe707d4e1 P ff28141737b8451a813f4524e704bbe7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:31.553887 2291 raft_consensus.cc:3037] T db2e0c4eb757451691f5873fe707d4e1 P 74f0ef49736744ceaccad487bd25d1cf [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:31.554662 2436 raft_consensus.cc:2454] T db2e0c4eb757451691f5873fe707d4e1 P ff28141737b8451a813f4524e704bbe7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02b0bac5d7944fb982708b589db96867 in term 1.
I20260619 14:37:31.554730 2291 raft_consensus.cc:2454] T db2e0c4eb757451691f5873fe707d4e1 P 74f0ef49736744ceaccad487bd25d1cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02b0bac5d7944fb982708b589db96867 in term 1.
I20260619 14:37:31.554901 2507 leader_election.cc:302] T db2e0c4eb757451691f5873fe707d4e1 P 02b0bac5d7944fb982708b589db96867 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02b0bac5d7944fb982708b589db96867, ff28141737b8451a813f4524e704bbe7; no voters:
I20260619 14:37:31.555058 2655 raft_consensus.cc:2781] T db2e0c4eb757451691f5873fe707d4e1 P 02b0bac5d7944fb982708b589db96867 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:37:31.555152 2655 raft_consensus.cc:686] T db2e0c4eb757451691f5873fe707d4e1 P 02b0bac5d7944fb982708b589db96867 [term 1 LEADER]: Becoming Leader. State: Replica: 02b0bac5d7944fb982708b589db96867, State: Running, Role: LEADER
I20260619 14:37:31.555284 2655 consensus_queue.cc:237] T db2e0c4eb757451691f5873fe707d4e1 P 02b0bac5d7944fb982708b589db96867 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f0ef49736744ceaccad487bd25d1cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33225 } } peers { permanent_uuid: "ff28141737b8451a813f4524e704bbe7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } } peers { permanent_uuid: "02b0bac5d7944fb982708b589db96867" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } }
I20260619 14:37:31.556792 2139 catalog_manager.cc:5495] T db2e0c4eb757451691f5873fe707d4e1 P 02b0bac5d7944fb982708b589db96867 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02b0bac5d7944fb982708b589db96867 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "02b0bac5d7944fb982708b589db96867" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f0ef49736744ceaccad487bd25d1cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff28141737b8451a813f4524e704bbe7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02b0bac5d7944fb982708b589db96867" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } health_report { overall_health: HEALTHY } } }
I20260619 14:37:31.582887 2436 raft_consensus.cc:1261] T db2e0c4eb757451691f5873fe707d4e1 P ff28141737b8451a813f4524e704bbe7 [term 1 FOLLOWER]: Refusing update from remote peer 02b0bac5d7944fb982708b589db96867: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:37:31.582922 2291 raft_consensus.cc:1261] T db2e0c4eb757451691f5873fe707d4e1 P 74f0ef49736744ceaccad487bd25d1cf [term 1 FOLLOWER]: Refusing update from remote peer 02b0bac5d7944fb982708b589db96867: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:37:31.583251 2689 consensus_queue.cc:1035] T db2e0c4eb757451691f5873fe707d4e1 P 02b0bac5d7944fb982708b589db96867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff28141737b8451a813f4524e704bbe7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:37:31.583405 2655 consensus_queue.cc:1035] T db2e0c4eb757451691f5873fe707d4e1 P 02b0bac5d7944fb982708b589db96867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74f0ef49736744ceaccad487bd25d1cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 14:37:31.595374 2674 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 14:37:33.144785 2139 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:37:33.144928 2139 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:37:33.147089 2139 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 96890e15464d40b1a3c6643bcb0abd9a: Sending DeleteTablet for 3 replicas of tablet db2e0c4eb757451691f5873fe707d4e1
I20260619 14:37:33.147563 2276 tablet_service.cc:1514] Processing DeleteTablet for tablet db2e0c4eb757451691f5873fe707d4e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:37:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53802
I20260619 14:37:33.147634 2416 tablet_service.cc:1514] Processing DeleteTablet for tablet db2e0c4eb757451691f5873fe707d4e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:37:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33156
I20260619 14:37:33.147668 2556 tablet_service.cc:1514] Processing DeleteTablet for tablet db2e0c4eb757451691f5873fe707d4e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:37:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45340
I20260619 14:37:33.147938 2728 tablet_replica.cc:330] T db2e0c4eb757451691f5873fe707d4e1 P ff28141737b8451a813f4524e704bbe7: stopping tablet replica
I20260619 14:37:33.148007 2729 tablet_replica.cc:330] T db2e0c4eb757451691f5873fe707d4e1 P 02b0bac5d7944fb982708b589db96867: stopping tablet replica
I20260619 14:37:33.148135 2729 raft_consensus.cc:2229] T db2e0c4eb757451691f5873fe707d4e1 P 02b0bac5d7944fb982708b589db96867 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:37:33.148138 2727 tablet_replica.cc:330] T db2e0c4eb757451691f5873fe707d4e1 P 74f0ef49736744ceaccad487bd25d1cf: stopping tablet replica
I20260619 14:37:33.148252 2727 raft_consensus.cc:2229] T db2e0c4eb757451691f5873fe707d4e1 P 74f0ef49736744ceaccad487bd25d1cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:37:33.148303 2729 raft_consensus.cc:2258] T db2e0c4eb757451691f5873fe707d4e1 P 02b0bac5d7944fb982708b589db96867 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:37:33.148355 2727 raft_consensus.cc:2258] T db2e0c4eb757451691f5873fe707d4e1 P 74f0ef49736744ceaccad487bd25d1cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:37:33.148526 2728 raft_consensus.cc:2229] T db2e0c4eb757451691f5873fe707d4e1 P ff28141737b8451a813f4524e704bbe7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:37:33.148631 2728 raft_consensus.cc:2258] T db2e0c4eb757451691f5873fe707d4e1 P ff28141737b8451a813f4524e704bbe7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:37:33.148975 2729 ts_tablet_manager.cc:1905] T db2e0c4eb757451691f5873fe707d4e1 P 02b0bac5d7944fb982708b589db96867: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:37:33.149085 2727 ts_tablet_manager.cc:1905] T db2e0c4eb757451691f5873fe707d4e1 P 74f0ef49736744ceaccad487bd25d1cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:37:33.151039 2729 ts_tablet_manager.cc:1918] T db2e0c4eb757451691f5873fe707d4e1 P 02b0bac5d7944fb982708b589db96867: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 14:37:33.151101 2729 log.cc:1196] T db2e0c4eb757451691f5873fe707d4e1 P 02b0bac5d7944fb982708b589db96867: Deleting WAL directory at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/ts-2/wal/wals/db2e0c4eb757451691f5873fe707d4e1
I20260619 14:37:33.151320 2729 ts_tablet_manager.cc:1939] T db2e0c4eb757451691f5873fe707d4e1 P 02b0bac5d7944fb982708b589db96867: Deleting consensus metadata
I20260619 14:37:33.151435 2727 ts_tablet_manager.cc:1918] T db2e0c4eb757451691f5873fe707d4e1 P 74f0ef49736744ceaccad487bd25d1cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 14:37:33.151497 2727 log.cc:1196] T db2e0c4eb757451691f5873fe707d4e1 P 74f0ef49736744ceaccad487bd25d1cf: Deleting WAL directory at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/ts-0/wal/wals/db2e0c4eb757451691f5873fe707d4e1
I20260619 14:37:33.151687 2727 ts_tablet_manager.cc:1939] T db2e0c4eb757451691f5873fe707d4e1 P 74f0ef49736744ceaccad487bd25d1cf: Deleting consensus metadata
I20260619 14:37:33.151878 2125 catalog_manager.cc:4841] TS 02b0bac5d7944fb982708b589db96867 (127.0.98.3:33147): tablet db2e0c4eb757451691f5873fe707d4e1 (table test-table [id=5ba2d5d51ad5488fa8a65d663d303457]) successfully deleted
I20260619 14:37:33.152233 2126 catalog_manager.cc:4841] TS 74f0ef49736744ceaccad487bd25d1cf (127.0.98.1:33225): tablet db2e0c4eb757451691f5873fe707d4e1 (table test-table [id=5ba2d5d51ad5488fa8a65d663d303457]) successfully deleted
I20260619 14:37:33.152460 2728 ts_tablet_manager.cc:1905] T db2e0c4eb757451691f5873fe707d4e1 P ff28141737b8451a813f4524e704bbe7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:37:33.154336 2728 ts_tablet_manager.cc:1918] T db2e0c4eb757451691f5873fe707d4e1 P ff28141737b8451a813f4524e704bbe7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 14:37:33.154399 2728 log.cc:1196] T db2e0c4eb757451691f5873fe707d4e1 P ff28141737b8451a813f4524e704bbe7: Deleting WAL directory at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/ts-1/wal/wals/db2e0c4eb757451691f5873fe707d4e1
I20260619 14:37:33.154587 2728 ts_tablet_manager.cc:1939] T db2e0c4eb757451691f5873fe707d4e1 P ff28141737b8451a813f4524e704bbe7: Deleting consensus metadata
I20260619 14:37:33.155094 2123 catalog_manager.cc:4841] TS ff28141737b8451a813f4524e704bbe7 (127.0.98.2:42505): tablet db2e0c4eb757451691f5873fe707d4e1 (table test-table [id=5ba2d5d51ad5488fa8a65d663d303457]) successfully deleted
Jun 19 14:37:33 dist-test-slave-32hj krb5kdc[2094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879853, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 14:37:33 dist-test-slave-32hj krb5kdc[2094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879853, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:37:33.174863 2737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:33.167154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45268 (local address 127.0.98.62:35971)
0619 14:37:33.167399 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:33.167408 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:33.167475 (+ 67us) server_negotiation.cc:407] Connection header received
0619 14:37:33.167610 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:33.167613 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:33.167666 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:33.167737 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:33.168436 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:33.169275 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:33.170063 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:33.170285 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:33.170358 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:33.172404 (+ 2046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:33.172407 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:33.172410 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:33.172432 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:33.173643 (+ 1211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:33.173992 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:33.173994 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:33.173995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:33.174063 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:33.174238 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:33.174240 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:33.174241 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:33.174394 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:33.174490 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:33.174636 (+ 146us) server_negotiation.cc:299] Negotiation successful
0619 14:37:33.174739 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":88,"threads_started":1}
I20260619 14:37:33.177058 2139 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45268:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:37:33.177464 2139 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:37:33.181643 2276 tablet_service.cc:1467] Processing CreateTablet for tablet 6362bfd477e148da9c9c228efcecfb05 (DEFAULT_TABLE table=test-table [id=1f893bd90c7941cbaf964988fea5a4da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:37:33.181762 2416 tablet_service.cc:1467] Processing CreateTablet for tablet 6362bfd477e148da9c9c228efcecfb05 (DEFAULT_TABLE table=test-table [id=1f893bd90c7941cbaf964988fea5a4da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:37:33.181759 2556 tablet_service.cc:1467] Processing CreateTablet for tablet 6362bfd477e148da9c9c228efcecfb05 (DEFAULT_TABLE table=test-table [id=1f893bd90c7941cbaf964988fea5a4da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:37:33.181917 2276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6362bfd477e148da9c9c228efcecfb05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:33.182019 2416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6362bfd477e148da9c9c228efcecfb05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:33.182018 2556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6362bfd477e148da9c9c228efcecfb05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:33.184046 2738 tablet_bootstrap.cc:492] T 6362bfd477e148da9c9c228efcecfb05 P 74f0ef49736744ceaccad487bd25d1cf: Bootstrap starting.
I20260619 14:37:33.184623 2739 tablet_bootstrap.cc:492] T 6362bfd477e148da9c9c228efcecfb05 P ff28141737b8451a813f4524e704bbe7: Bootstrap starting.
I20260619 14:37:33.184652 2740 tablet_bootstrap.cc:492] T 6362bfd477e148da9c9c228efcecfb05 P 02b0bac5d7944fb982708b589db96867: Bootstrap starting.
I20260619 14:37:33.185086 2738 tablet_bootstrap.cc:654] T 6362bfd477e148da9c9c228efcecfb05 P 74f0ef49736744ceaccad487bd25d1cf: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:33.185458 2740 tablet_bootstrap.cc:654] T 6362bfd477e148da9c9c228efcecfb05 P 02b0bac5d7944fb982708b589db96867: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:33.185468 2739 tablet_bootstrap.cc:654] T 6362bfd477e148da9c9c228efcecfb05 P ff28141737b8451a813f4524e704bbe7: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:33.186089 2738 tablet_bootstrap.cc:492] T 6362bfd477e148da9c9c228efcecfb05 P 74f0ef49736744ceaccad487bd25d1cf: No bootstrap required, opened a new log
I20260619 14:37:33.186193 2738 ts_tablet_manager.cc:1397] T 6362bfd477e148da9c9c228efcecfb05 P 74f0ef49736744ceaccad487bd25d1cf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 14:37:33.186581 2740 tablet_bootstrap.cc:492] T 6362bfd477e148da9c9c228efcecfb05 P 02b0bac5d7944fb982708b589db96867: No bootstrap required, opened a new log
I20260619 14:37:33.186666 2740 ts_tablet_manager.cc:1397] T 6362bfd477e148da9c9c228efcecfb05 P 02b0bac5d7944fb982708b589db96867: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 14:37:33.186630 2738 raft_consensus.cc:348] T 6362bfd477e148da9c9c228efcecfb05 P 74f0ef49736744ceaccad487bd25d1cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f0ef49736744ceaccad487bd25d1cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33225 } } peers { permanent_uuid: "ff28141737b8451a813f4524e704bbe7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } } peers { permanent_uuid: "02b0bac5d7944fb982708b589db96867" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } }
I20260619 14:37:33.186748 2738 raft_consensus.cc:374] T 6362bfd477e148da9c9c228efcecfb05 P 74f0ef49736744ceaccad487bd25d1cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:33.186785 2738 raft_consensus.cc:729] T 6362bfd477e148da9c9c228efcecfb05 P 74f0ef49736744ceaccad487bd25d1cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74f0ef49736744ceaccad487bd25d1cf, State: Initialized, Role: FOLLOWER
I20260619 14:37:33.186936 2739 tablet_bootstrap.cc:492] T 6362bfd477e148da9c9c228efcecfb05 P ff28141737b8451a813f4524e704bbe7: No bootstrap required, opened a new log
I20260619 14:37:33.186910 2738 consensus_queue.cc:260] T 6362bfd477e148da9c9c228efcecfb05 P 74f0ef49736744ceaccad487bd25d1cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f0ef49736744ceaccad487bd25d1cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33225 } } peers { permanent_uuid: "ff28141737b8451a813f4524e704bbe7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } } peers { permanent_uuid: "02b0bac5d7944fb982708b589db96867" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } }
I20260619 14:37:33.187012 2739 ts_tablet_manager.cc:1397] T 6362bfd477e148da9c9c228efcecfb05 P ff28141737b8451a813f4524e704bbe7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 14:37:33.187044 2740 raft_consensus.cc:348] T 6362bfd477e148da9c9c228efcecfb05 P 02b0bac5d7944fb982708b589db96867 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f0ef49736744ceaccad487bd25d1cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33225 } } peers { permanent_uuid: "ff28141737b8451a813f4524e704bbe7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } } peers { permanent_uuid: "02b0bac5d7944fb982708b589db96867" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } }
I20260619 14:37:33.187146 2740 raft_consensus.cc:374] T 6362bfd477e148da9c9c228efcecfb05 P 02b0bac5d7944fb982708b589db96867 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:33.187193 2740 raft_consensus.cc:729] T 6362bfd477e148da9c9c228efcecfb05 P 02b0bac5d7944fb982708b589db96867 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02b0bac5d7944fb982708b589db96867, State: Initialized, Role: FOLLOWER
I20260619 14:37:33.187196 2738 ts_tablet_manager.cc:1428] T 6362bfd477e148da9c9c228efcecfb05 P 74f0ef49736744ceaccad487bd25d1cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:37:33.187314 2740 consensus_queue.cc:260] T 6362bfd477e148da9c9c228efcecfb05 P 02b0bac5d7944fb982708b589db96867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f0ef49736744ceaccad487bd25d1cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33225 } } peers { permanent_uuid: "ff28141737b8451a813f4524e704bbe7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } } peers { permanent_uuid: "02b0bac5d7944fb982708b589db96867" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } }
I20260619 14:37:33.187417 2739 raft_consensus.cc:348] T 6362bfd477e148da9c9c228efcecfb05 P ff28141737b8451a813f4524e704bbe7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f0ef49736744ceaccad487bd25d1cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33225 } } peers { permanent_uuid: "ff28141737b8451a813f4524e704bbe7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } } peers { permanent_uuid: "02b0bac5d7944fb982708b589db96867" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } }
I20260619 14:37:33.187507 2739 raft_consensus.cc:374] T 6362bfd477e148da9c9c228efcecfb05 P ff28141737b8451a813f4524e704bbe7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:33.187533 2739 raft_consensus.cc:729] T 6362bfd477e148da9c9c228efcecfb05 P ff28141737b8451a813f4524e704bbe7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff28141737b8451a813f4524e704bbe7, State: Initialized, Role: FOLLOWER
I20260619 14:37:33.187674 2739 consensus_queue.cc:260] T 6362bfd477e148da9c9c228efcecfb05 P ff28141737b8451a813f4524e704bbe7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f0ef49736744ceaccad487bd25d1cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33225 } } peers { permanent_uuid: "ff28141737b8451a813f4524e704bbe7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } } peers { permanent_uuid: "02b0bac5d7944fb982708b589db96867" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } }
I20260619 14:37:33.188048 2740 ts_tablet_manager.cc:1428] T 6362bfd477e148da9c9c228efcecfb05 P 02b0bac5d7944fb982708b589db96867: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:37:33.188565 2739 ts_tablet_manager.cc:1428] T 6362bfd477e148da9c9c228efcecfb05 P ff28141737b8451a813f4524e704bbe7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260619 14:37:33.261199 2679 raft_consensus.cc:482] T 6362bfd477e148da9c9c228efcecfb05 P 74f0ef49736744ceaccad487bd25d1cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:37:33.261327 2679 raft_consensus.cc:504] T 6362bfd477e148da9c9c228efcecfb05 P 74f0ef49736744ceaccad487bd25d1cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f0ef49736744ceaccad487bd25d1cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33225 } } peers { permanent_uuid: "ff28141737b8451a813f4524e704bbe7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } } peers { permanent_uuid: "02b0bac5d7944fb982708b589db96867" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } }
I20260619 14:37:33.261718 2679 leader_election.cc:288] T 6362bfd477e148da9c9c228efcecfb05 P 74f0ef49736744ceaccad487bd25d1cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff28141737b8451a813f4524e704bbe7 (127.0.98.2:42505), 02b0bac5d7944fb982708b589db96867 (127.0.98.3:33147)
I20260619 14:37:33.262089 2436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6362bfd477e148da9c9c228efcecfb05" candidate_uuid: "74f0ef49736744ceaccad487bd25d1cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff28141737b8451a813f4524e704bbe7" is_pre_election: true
I20260619 14:37:33.262115 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6362bfd477e148da9c9c228efcecfb05" candidate_uuid: "74f0ef49736744ceaccad487bd25d1cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02b0bac5d7944fb982708b589db96867" is_pre_election: true
I20260619 14:37:33.262234 2576 raft_consensus.cc:2454] T 6362bfd477e148da9c9c228efcecfb05 P 02b0bac5d7944fb982708b589db96867 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74f0ef49736744ceaccad487bd25d1cf in term 0.
I20260619 14:37:33.262247 2436 raft_consensus.cc:2454] T 6362bfd477e148da9c9c228efcecfb05 P ff28141737b8451a813f4524e704bbe7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74f0ef49736744ceaccad487bd25d1cf in term 0.
I20260619 14:37:33.262557 2229 leader_election.cc:302] T 6362bfd477e148da9c9c228efcecfb05 P 74f0ef49736744ceaccad487bd25d1cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 02b0bac5d7944fb982708b589db96867, 74f0ef49736744ceaccad487bd25d1cf, ff28141737b8451a813f4524e704bbe7; no voters:
I20260619 14:37:33.262735 2679 raft_consensus.cc:2781] T 6362bfd477e148da9c9c228efcecfb05 P 74f0ef49736744ceaccad487bd25d1cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:37:33.262791 2679 raft_consensus.cc:482] T 6362bfd477e148da9c9c228efcecfb05 P 74f0ef49736744ceaccad487bd25d1cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:37:33.262817 2679 raft_consensus.cc:3037] T 6362bfd477e148da9c9c228efcecfb05 P 74f0ef49736744ceaccad487bd25d1cf [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:33.263782 2679 raft_consensus.cc:504] T 6362bfd477e148da9c9c228efcecfb05 P 74f0ef49736744ceaccad487bd25d1cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f0ef49736744ceaccad487bd25d1cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33225 } } peers { permanent_uuid: "ff28141737b8451a813f4524e704bbe7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } } peers { permanent_uuid: "02b0bac5d7944fb982708b589db96867" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } }
I20260619 14:37:33.264070 2679 leader_election.cc:288] T 6362bfd477e148da9c9c228efcecfb05 P 74f0ef49736744ceaccad487bd25d1cf [CANDIDATE]: Term 1 election: Requested vote from peers ff28141737b8451a813f4524e704bbe7 (127.0.98.2:42505), 02b0bac5d7944fb982708b589db96867 (127.0.98.3:33147)
I20260619 14:37:33.264392 2436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6362bfd477e148da9c9c228efcecfb05" candidate_uuid: "74f0ef49736744ceaccad487bd25d1cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff28141737b8451a813f4524e704bbe7"
I20260619 14:37:33.264392 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6362bfd477e148da9c9c228efcecfb05" candidate_uuid: "74f0ef49736744ceaccad487bd25d1cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02b0bac5d7944fb982708b589db96867"
I20260619 14:37:33.264513 2576 raft_consensus.cc:3037] T 6362bfd477e148da9c9c228efcecfb05 P 02b0bac5d7944fb982708b589db96867 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:33.264513 2436 raft_consensus.cc:3037] T 6362bfd477e148da9c9c228efcecfb05 P ff28141737b8451a813f4524e704bbe7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:33.265621 2436 raft_consensus.cc:2454] T 6362bfd477e148da9c9c228efcecfb05 P ff28141737b8451a813f4524e704bbe7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74f0ef49736744ceaccad487bd25d1cf in term 1.
I20260619 14:37:33.265615 2576 raft_consensus.cc:2454] T 6362bfd477e148da9c9c228efcecfb05 P 02b0bac5d7944fb982708b589db96867 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74f0ef49736744ceaccad487bd25d1cf in term 1.
I20260619 14:37:33.265887 2229 leader_election.cc:302] T 6362bfd477e148da9c9c228efcecfb05 P 74f0ef49736744ceaccad487bd25d1cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02b0bac5d7944fb982708b589db96867, 74f0ef49736744ceaccad487bd25d1cf; no voters:
I20260619 14:37:33.266033 2679 raft_consensus.cc:2781] T 6362bfd477e148da9c9c228efcecfb05 P 74f0ef49736744ceaccad487bd25d1cf [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:37:33.266117 2679 raft_consensus.cc:686] T 6362bfd477e148da9c9c228efcecfb05 P 74f0ef49736744ceaccad487bd25d1cf [term 1 LEADER]: Becoming Leader. State: Replica: 74f0ef49736744ceaccad487bd25d1cf, State: Running, Role: LEADER
I20260619 14:37:33.266265 2679 consensus_queue.cc:237] T 6362bfd477e148da9c9c228efcecfb05 P 74f0ef49736744ceaccad487bd25d1cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f0ef49736744ceaccad487bd25d1cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33225 } } peers { permanent_uuid: "ff28141737b8451a813f4524e704bbe7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } } peers { permanent_uuid: "02b0bac5d7944fb982708b589db96867" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } }
I20260619 14:37:33.267786 2139 catalog_manager.cc:5495] T 6362bfd477e148da9c9c228efcecfb05 P 74f0ef49736744ceaccad487bd25d1cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 74f0ef49736744ceaccad487bd25d1cf (127.0.98.1). New cstate: current_term: 1 leader_uuid: "74f0ef49736744ceaccad487bd25d1cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f0ef49736744ceaccad487bd25d1cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff28141737b8451a813f4524e704bbe7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02b0bac5d7944fb982708b589db96867" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } health_report { overall_health: UNKNOWN } } }
I20260619 14:37:33.287607 2747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:33.284900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53828 (local address 127.0.98.1:33225)
0619 14:37:33.285130 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:33.285136 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:33.285147 (+ 11us) server_negotiation.cc:407] Connection header received
0619 14:37:33.285199 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:33.285201 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:33.285240 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:33.285305 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:37:33.285493 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:33.285993 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:33.286735 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:33.287010 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:33.287082 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:33.287296 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:37:33.287359 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:37:33.287404 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:33.287463 (+ 59us) server_negotiation.cc:299] Negotiation successful
0619 14:37:33.287500 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":96,"threads_started":1}
I20260619 14:37:33.295270 2750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:33.292436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33225 (local address 127.0.98.1:60623)
0619 14:37:33.292668 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:33.292699 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:33.292800 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:33.292988 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:33.292990 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:33.292996 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:37:33.293087 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:33.293092 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:33.293730 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:33.293733 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:33.294312 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:33.294320 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:33.294700 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:33.294703 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:33.294786 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:33.294806 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:37:33.295019 (+ 213us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:37:33.295025 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 14:37:33.295080 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 14:37:33.295150 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":87,"threads_started":1}
I20260619 14:37:33.295270 2747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:33.292521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60623 (local address 127.0.98.1:33225)
0619 14:37:33.292644 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:33.292648 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:33.292776 (+ 128us) server_negotiation.cc:407] Connection header received
0619 14:37:33.292869 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:33.292872 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:33.292909 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:33.292964 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:37:33.293171 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:33.293637 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:33.294412 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:33.294613 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:33.294690 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:33.294897 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:37:33.294949 (+ 52us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:37:33.295005 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:33.295086 (+ 81us) server_negotiation.cc:299] Negotiation successful
0619 14:37:33.295137 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":55}
I20260619 14:37:33.299989 2576 raft_consensus.cc:1261] T 6362bfd477e148da9c9c228efcecfb05 P 02b0bac5d7944fb982708b589db96867 [term 1 FOLLOWER]: Refusing update from remote peer 74f0ef49736744ceaccad487bd25d1cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:37:33.299993 2436 raft_consensus.cc:1261] T 6362bfd477e148da9c9c228efcecfb05 P ff28141737b8451a813f4524e704bbe7 [term 1 FOLLOWER]: Refusing update from remote peer 74f0ef49736744ceaccad487bd25d1cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:37:33.300315 2713 consensus_queue.cc:1035] T 6362bfd477e148da9c9c228efcecfb05 P 74f0ef49736744ceaccad487bd25d1cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff28141737b8451a813f4524e704bbe7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:37:33.300474 2708 consensus_queue.cc:1035] T 6362bfd477e148da9c9c228efcecfb05 P 74f0ef49736744ceaccad487bd25d1cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "02b0bac5d7944fb982708b589db96867" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:37:33.329813 2137 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:37:33.329954 2137 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:37:33.331743 2137 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 96890e15464d40b1a3c6643bcb0abd9a: Sending DeleteTablet for 3 replicas of tablet 6362bfd477e148da9c9c228efcecfb05
I20260619 14:37:33.332180 2275 tablet_service.cc:1514] Processing DeleteTablet for tablet 6362bfd477e148da9c9c228efcecfb05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:37:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53802
I20260619 14:37:33.332290 2416 tablet_service.cc:1514] Processing DeleteTablet for tablet 6362bfd477e148da9c9c228efcecfb05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:37:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33156
I20260619 14:37:33.332347 2727 tablet_replica.cc:330] T 6362bfd477e148da9c9c228efcecfb05 P 74f0ef49736744ceaccad487bd25d1cf: stopping tablet replica
I20260619 14:37:33.332362 2556 tablet_service.cc:1514] Processing DeleteTablet for tablet 6362bfd477e148da9c9c228efcecfb05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:37:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45340
I20260619 14:37:33.332454 2728 tablet_replica.cc:330] T 6362bfd477e148da9c9c228efcecfb05 P ff28141737b8451a813f4524e704bbe7: stopping tablet replica
I20260619 14:37:33.332454 2727 raft_consensus.cc:2229] T 6362bfd477e148da9c9c228efcecfb05 P 74f0ef49736744ceaccad487bd25d1cf [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:37:33.332496 2729 tablet_replica.cc:330] T 6362bfd477e148da9c9c228efcecfb05 P 02b0bac5d7944fb982708b589db96867: stopping tablet replica
I20260619 14:37:33.332551 2728 raft_consensus.cc:2229] T 6362bfd477e148da9c9c228efcecfb05 P ff28141737b8451a813f4524e704bbe7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:37:33.332580 2729 raft_consensus.cc:2229] T 6362bfd477e148da9c9c228efcecfb05 P 02b0bac5d7944fb982708b589db96867 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:37:33.332643 2728 raft_consensus.cc:2258] T 6362bfd477e148da9c9c228efcecfb05 P ff28141737b8451a813f4524e704bbe7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:37:33.332670 2729 raft_consensus.cc:2258] T 6362bfd477e148da9c9c228efcecfb05 P 02b0bac5d7944fb982708b589db96867 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:37:33.332762 2727 raft_consensus.cc:2258] T 6362bfd477e148da9c9c228efcecfb05 P 74f0ef49736744ceaccad487bd25d1cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:37:33.333684 2729 ts_tablet_manager.cc:1905] T 6362bfd477e148da9c9c228efcecfb05 P 02b0bac5d7944fb982708b589db96867: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:37:33.333696 2728 ts_tablet_manager.cc:1905] T 6362bfd477e148da9c9c228efcecfb05 P ff28141737b8451a813f4524e704bbe7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:37:33.336076 2728 ts_tablet_manager.cc:1918] T 6362bfd477e148da9c9c228efcecfb05 P ff28141737b8451a813f4524e704bbe7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:37:33.336100 2729 ts_tablet_manager.cc:1918] T 6362bfd477e148da9c9c228efcecfb05 P 02b0bac5d7944fb982708b589db96867: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:37:33.336154 2729 log.cc:1196] T 6362bfd477e148da9c9c228efcecfb05 P 02b0bac5d7944fb982708b589db96867: Deleting WAL directory at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/ts-2/wal/wals/6362bfd477e148da9c9c228efcecfb05
I20260619 14:37:33.336154 2728 log.cc:1196] T 6362bfd477e148da9c9c228efcecfb05 P ff28141737b8451a813f4524e704bbe7: Deleting WAL directory at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/ts-1/wal/wals/6362bfd477e148da9c9c228efcecfb05
I20260619 14:37:33.336385 2728 ts_tablet_manager.cc:1939] T 6362bfd477e148da9c9c228efcecfb05 P ff28141737b8451a813f4524e704bbe7: Deleting consensus metadata
I20260619 14:37:33.336406 2729 ts_tablet_manager.cc:1939] T 6362bfd477e148da9c9c228efcecfb05 P 02b0bac5d7944fb982708b589db96867: Deleting consensus metadata
I20260619 14:37:33.337038 2123 catalog_manager.cc:4841] TS ff28141737b8451a813f4524e704bbe7 (127.0.98.2:42505): tablet 6362bfd477e148da9c9c228efcecfb05 (table test-table [id=1f893bd90c7941cbaf964988fea5a4da]) successfully deleted
I20260619 14:37:33.337260 2727 ts_tablet_manager.cc:1905] T 6362bfd477e148da9c9c228efcecfb05 P 74f0ef49736744ceaccad487bd25d1cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:37:33.338581 2137 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45268:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:37:33.339011 2137 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:37:33.339612 2727 ts_tablet_manager.cc:1918] T 6362bfd477e148da9c9c228efcecfb05 P 74f0ef49736744ceaccad487bd25d1cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:37:33.339675 2727 log.cc:1196] T 6362bfd477e148da9c9c228efcecfb05 P 74f0ef49736744ceaccad487bd25d1cf: Deleting WAL directory at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/ts-0/wal/wals/6362bfd477e148da9c9c228efcecfb05
I20260619 14:37:33.339886 2727 ts_tablet_manager.cc:1939] T 6362bfd477e148da9c9c228efcecfb05 P 74f0ef49736744ceaccad487bd25d1cf: Deleting consensus metadata
I20260619 14:37:33.340181 2125 catalog_manager.cc:4841] TS 02b0bac5d7944fb982708b589db96867 (127.0.98.3:33147): tablet 6362bfd477e148da9c9c228efcecfb05 (table test-table [id=1f893bd90c7941cbaf964988fea5a4da]) successfully deleted
I20260619 14:37:33.340464 2126 catalog_manager.cc:4841] TS 74f0ef49736744ceaccad487bd25d1cf (127.0.98.1:33225): tablet 6362bfd477e148da9c9c228efcecfb05 (table test-table [id=1f893bd90c7941cbaf964988fea5a4da]) successfully deleted
I20260619 14:37:33.343851 2416 tablet_service.cc:1467] Processing CreateTablet for tablet e9643e74729946b4afbe133f4d71cb36 (DEFAULT_TABLE table=test-table [id=5bfb8f4a10bb4e879f1121322708379f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:37:33.343851 2275 tablet_service.cc:1467] Processing CreateTablet for tablet e9643e74729946b4afbe133f4d71cb36 (DEFAULT_TABLE table=test-table [id=5bfb8f4a10bb4e879f1121322708379f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:37:33.344094 2416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9643e74729946b4afbe133f4d71cb36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:33.344094 2275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9643e74729946b4afbe133f4d71cb36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:33.346153 2738 tablet_bootstrap.cc:492] T e9643e74729946b4afbe133f4d71cb36 P 74f0ef49736744ceaccad487bd25d1cf: Bootstrap starting.
I20260619 14:37:33.346747 2738 tablet_bootstrap.cc:654] T e9643e74729946b4afbe133f4d71cb36 P 74f0ef49736744ceaccad487bd25d1cf: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:33.347160 2739 tablet_bootstrap.cc:492] T e9643e74729946b4afbe133f4d71cb36 P ff28141737b8451a813f4524e704bbe7: Bootstrap starting.
I20260619 14:37:33.347831 2739 tablet_bootstrap.cc:654] T e9643e74729946b4afbe133f4d71cb36 P ff28141737b8451a813f4524e704bbe7: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:33.348415 2556 tablet_service.cc:1467] Processing CreateTablet for tablet e9643e74729946b4afbe133f4d71cb36 (DEFAULT_TABLE table=test-table [id=5bfb8f4a10bb4e879f1121322708379f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:37:33.348616 2556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9643e74729946b4afbe133f4d71cb36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:33.349282 2738 tablet_bootstrap.cc:492] T e9643e74729946b4afbe133f4d71cb36 P 74f0ef49736744ceaccad487bd25d1cf: No bootstrap required, opened a new log
I20260619 14:37:33.349365 2738 ts_tablet_manager.cc:1397] T e9643e74729946b4afbe133f4d71cb36 P 74f0ef49736744ceaccad487bd25d1cf: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 14:37:33.349726 2738 raft_consensus.cc:348] T e9643e74729946b4afbe133f4d71cb36 P 74f0ef49736744ceaccad487bd25d1cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f0ef49736744ceaccad487bd25d1cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33225 } } peers { permanent_uuid: "ff28141737b8451a813f4524e704bbe7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } } peers { permanent_uuid: "02b0bac5d7944fb982708b589db96867" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } }
I20260619 14:37:33.349833 2738 raft_consensus.cc:374] T e9643e74729946b4afbe133f4d71cb36 P 74f0ef49736744ceaccad487bd25d1cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:33.349861 2738 raft_consensus.cc:729] T e9643e74729946b4afbe133f4d71cb36 P 74f0ef49736744ceaccad487bd25d1cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74f0ef49736744ceaccad487bd25d1cf, State: Initialized, Role: FOLLOWER
I20260619 14:37:33.349972 2738 consensus_queue.cc:260] T e9643e74729946b4afbe133f4d71cb36 P 74f0ef49736744ceaccad487bd25d1cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f0ef49736744ceaccad487bd25d1cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33225 } } peers { permanent_uuid: "ff28141737b8451a813f4524e704bbe7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } } peers { permanent_uuid: "02b0bac5d7944fb982708b589db96867" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } }
I20260619 14:37:33.350206 2738 ts_tablet_manager.cc:1428] T e9643e74729946b4afbe133f4d71cb36 P 74f0ef49736744ceaccad487bd25d1cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:37:33.350416 2740 tablet_bootstrap.cc:492] T e9643e74729946b4afbe133f4d71cb36 P 02b0bac5d7944fb982708b589db96867: Bootstrap starting.
I20260619 14:37:33.350688 2739 tablet_bootstrap.cc:492] T e9643e74729946b4afbe133f4d71cb36 P ff28141737b8451a813f4524e704bbe7: No bootstrap required, opened a new log
I20260619 14:37:33.350759 2739 ts_tablet_manager.cc:1397] T e9643e74729946b4afbe133f4d71cb36 P ff28141737b8451a813f4524e704bbe7: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260619 14:37:33.351089 2740 tablet_bootstrap.cc:654] T e9643e74729946b4afbe133f4d71cb36 P 02b0bac5d7944fb982708b589db96867: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:33.351135 2739 raft_consensus.cc:348] T e9643e74729946b4afbe133f4d71cb36 P ff28141737b8451a813f4524e704bbe7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f0ef49736744ceaccad487bd25d1cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33225 } } peers { permanent_uuid: "ff28141737b8451a813f4524e704bbe7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } } peers { permanent_uuid: "02b0bac5d7944fb982708b589db96867" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } }
I20260619 14:37:33.351227 2739 raft_consensus.cc:374] T e9643e74729946b4afbe133f4d71cb36 P ff28141737b8451a813f4524e704bbe7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:33.351255 2739 raft_consensus.cc:729] T e9643e74729946b4afbe133f4d71cb36 P ff28141737b8451a813f4524e704bbe7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff28141737b8451a813f4524e704bbe7, State: Initialized, Role: FOLLOWER
I20260619 14:37:33.351362 2739 consensus_queue.cc:260] T e9643e74729946b4afbe133f4d71cb36 P ff28141737b8451a813f4524e704bbe7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f0ef49736744ceaccad487bd25d1cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33225 } } peers { permanent_uuid: "ff28141737b8451a813f4524e704bbe7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } } peers { permanent_uuid: "02b0bac5d7944fb982708b589db96867" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } }
I20260619 14:37:33.351594 2739 ts_tablet_manager.cc:1428] T e9643e74729946b4afbe133f4d71cb36 P ff28141737b8451a813f4524e704bbe7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:37:33.351946 2740 tablet_bootstrap.cc:492] T e9643e74729946b4afbe133f4d71cb36 P 02b0bac5d7944fb982708b589db96867: No bootstrap required, opened a new log
I20260619 14:37:33.352022 2740 ts_tablet_manager.cc:1397] T e9643e74729946b4afbe133f4d71cb36 P 02b0bac5d7944fb982708b589db96867: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 14:37:33.352388 2740 raft_consensus.cc:348] T e9643e74729946b4afbe133f4d71cb36 P 02b0bac5d7944fb982708b589db96867 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f0ef49736744ceaccad487bd25d1cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33225 } } peers { permanent_uuid: "ff28141737b8451a813f4524e704bbe7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } } peers { permanent_uuid: "02b0bac5d7944fb982708b589db96867" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } }
I20260619 14:37:33.352492 2740 raft_consensus.cc:374] T e9643e74729946b4afbe133f4d71cb36 P 02b0bac5d7944fb982708b589db96867 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:33.352522 2740 raft_consensus.cc:729] T e9643e74729946b4afbe133f4d71cb36 P 02b0bac5d7944fb982708b589db96867 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02b0bac5d7944fb982708b589db96867, State: Initialized, Role: FOLLOWER
I20260619 14:37:33.352638 2740 consensus_queue.cc:260] T e9643e74729946b4afbe133f4d71cb36 P 02b0bac5d7944fb982708b589db96867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f0ef49736744ceaccad487bd25d1cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33225 } } peers { permanent_uuid: "ff28141737b8451a813f4524e704bbe7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } } peers { permanent_uuid: "02b0bac5d7944fb982708b589db96867" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } }
I20260619 14:37:33.352974 2740 ts_tablet_manager.cc:1428] T e9643e74729946b4afbe133f4d71cb36 P 02b0bac5d7944fb982708b589db96867: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:37:33.408272 2708 raft_consensus.cc:482] T e9643e74729946b4afbe133f4d71cb36 P 74f0ef49736744ceaccad487bd25d1cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:37:33.408404 2708 raft_consensus.cc:504] T e9643e74729946b4afbe133f4d71cb36 P 74f0ef49736744ceaccad487bd25d1cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f0ef49736744ceaccad487bd25d1cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33225 } } peers { permanent_uuid: "ff28141737b8451a813f4524e704bbe7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } } peers { permanent_uuid: "02b0bac5d7944fb982708b589db96867" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } }
I20260619 14:37:33.408838 2708 leader_election.cc:288] T e9643e74729946b4afbe133f4d71cb36 P 74f0ef49736744ceaccad487bd25d1cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff28141737b8451a813f4524e704bbe7 (127.0.98.2:42505), 02b0bac5d7944fb982708b589db96867 (127.0.98.3:33147)
I20260619 14:37:33.409228 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9643e74729946b4afbe133f4d71cb36" candidate_uuid: "74f0ef49736744ceaccad487bd25d1cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02b0bac5d7944fb982708b589db96867" is_pre_election: true
I20260619 14:37:33.409245 2436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9643e74729946b4afbe133f4d71cb36" candidate_uuid: "74f0ef49736744ceaccad487bd25d1cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff28141737b8451a813f4524e704bbe7" is_pre_election: true
I20260619 14:37:33.409369 2575 raft_consensus.cc:2454] T e9643e74729946b4afbe133f4d71cb36 P 02b0bac5d7944fb982708b589db96867 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74f0ef49736744ceaccad487bd25d1cf in term 0.
I20260619 14:37:33.409427 2436 raft_consensus.cc:2454] T e9643e74729946b4afbe133f4d71cb36 P ff28141737b8451a813f4524e704bbe7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74f0ef49736744ceaccad487bd25d1cf in term 0.
I20260619 14:37:33.409716 2229 leader_election.cc:302] T e9643e74729946b4afbe133f4d71cb36 P 74f0ef49736744ceaccad487bd25d1cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 02b0bac5d7944fb982708b589db96867, 74f0ef49736744ceaccad487bd25d1cf, ff28141737b8451a813f4524e704bbe7; no voters:
I20260619 14:37:33.409891 2708 raft_consensus.cc:2781] T e9643e74729946b4afbe133f4d71cb36 P 74f0ef49736744ceaccad487bd25d1cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:37:33.409957 2708 raft_consensus.cc:482] T e9643e74729946b4afbe133f4d71cb36 P 74f0ef49736744ceaccad487bd25d1cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:37:33.410004 2708 raft_consensus.cc:3037] T e9643e74729946b4afbe133f4d71cb36 P 74f0ef49736744ceaccad487bd25d1cf [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:33.410882 2708 raft_consensus.cc:504] T e9643e74729946b4afbe133f4d71cb36 P 74f0ef49736744ceaccad487bd25d1cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f0ef49736744ceaccad487bd25d1cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33225 } } peers { permanent_uuid: "ff28141737b8451a813f4524e704bbe7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } } peers { permanent_uuid: "02b0bac5d7944fb982708b589db96867" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } }
I20260619 14:37:33.411202 2708 leader_election.cc:288] T e9643e74729946b4afbe133f4d71cb36 P 74f0ef49736744ceaccad487bd25d1cf [CANDIDATE]: Term 1 election: Requested vote from peers ff28141737b8451a813f4524e704bbe7 (127.0.98.2:42505), 02b0bac5d7944fb982708b589db96867 (127.0.98.3:33147)
I20260619 14:37:33.411530 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9643e74729946b4afbe133f4d71cb36" candidate_uuid: "74f0ef49736744ceaccad487bd25d1cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02b0bac5d7944fb982708b589db96867"
I20260619 14:37:33.411530 2436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9643e74729946b4afbe133f4d71cb36" candidate_uuid: "74f0ef49736744ceaccad487bd25d1cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff28141737b8451a813f4524e704bbe7"
I20260619 14:37:33.411640 2436 raft_consensus.cc:3037] T e9643e74729946b4afbe133f4d71cb36 P ff28141737b8451a813f4524e704bbe7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:33.411641 2575 raft_consensus.cc:3037] T e9643e74729946b4afbe133f4d71cb36 P 02b0bac5d7944fb982708b589db96867 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:33.412665 2436 raft_consensus.cc:2454] T e9643e74729946b4afbe133f4d71cb36 P ff28141737b8451a813f4524e704bbe7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74f0ef49736744ceaccad487bd25d1cf in term 1.
I20260619 14:37:33.412698 2575 raft_consensus.cc:2454] T e9643e74729946b4afbe133f4d71cb36 P 02b0bac5d7944fb982708b589db96867 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74f0ef49736744ceaccad487bd25d1cf in term 1.
I20260619 14:37:33.412958 2229 leader_election.cc:302] T e9643e74729946b4afbe133f4d71cb36 P 74f0ef49736744ceaccad487bd25d1cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02b0bac5d7944fb982708b589db96867, 74f0ef49736744ceaccad487bd25d1cf; no voters:
I20260619 14:37:33.413201 2708 raft_consensus.cc:2781] T e9643e74729946b4afbe133f4d71cb36 P 74f0ef49736744ceaccad487bd25d1cf [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:37:33.413496 2708 raft_consensus.cc:686] T e9643e74729946b4afbe133f4d71cb36 P 74f0ef49736744ceaccad487bd25d1cf [term 1 LEADER]: Becoming Leader. State: Replica: 74f0ef49736744ceaccad487bd25d1cf, State: Running, Role: LEADER
I20260619 14:37:33.413653 2708 consensus_queue.cc:237] T e9643e74729946b4afbe133f4d71cb36 P 74f0ef49736744ceaccad487bd25d1cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f0ef49736744ceaccad487bd25d1cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33225 } } peers { permanent_uuid: "ff28141737b8451a813f4524e704bbe7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } } peers { permanent_uuid: "02b0bac5d7944fb982708b589db96867" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } }
I20260619 14:37:33.415153 2137 catalog_manager.cc:5495] T e9643e74729946b4afbe133f4d71cb36 P 74f0ef49736744ceaccad487bd25d1cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 74f0ef49736744ceaccad487bd25d1cf (127.0.98.1). New cstate: current_term: 1 leader_uuid: "74f0ef49736744ceaccad487bd25d1cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74f0ef49736744ceaccad487bd25d1cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff28141737b8451a813f4524e704bbe7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02b0bac5d7944fb982708b589db96867" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 } health_report { overall_health: UNKNOWN } } }
I20260619 14:37:33.444949 2575 raft_consensus.cc:1261] T e9643e74729946b4afbe133f4d71cb36 P 02b0bac5d7944fb982708b589db96867 [term 1 FOLLOWER]: Refusing update from remote peer 74f0ef49736744ceaccad487bd25d1cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:37:33.445039 2436 raft_consensus.cc:1261] T e9643e74729946b4afbe133f4d71cb36 P ff28141737b8451a813f4524e704bbe7 [term 1 FOLLOWER]: Refusing update from remote peer 74f0ef49736744ceaccad487bd25d1cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:37:33.445336 2708 consensus_queue.cc:1035] T e9643e74729946b4afbe133f4d71cb36 P 74f0ef49736744ceaccad487bd25d1cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "02b0bac5d7944fb982708b589db96867" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:37:33.445586 2679 consensus_queue.cc:1035] T e9643e74729946b4afbe133f4d71cb36 P 74f0ef49736744ceaccad487bd25d1cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff28141737b8451a813f4524e704bbe7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 14:37:33.460933 2706 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 14:37:35.017653 2137 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:37:35.017793 2137 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:37:35.019943 2137 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 96890e15464d40b1a3c6643bcb0abd9a: Sending DeleteTablet for 3 replicas of tablet e9643e74729946b4afbe133f4d71cb36
I20260619 14:37:35.020394 2275 tablet_service.cc:1514] Processing DeleteTablet for tablet e9643e74729946b4afbe133f4d71cb36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:37:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53802
I20260619 14:37:35.020510 2416 tablet_service.cc:1514] Processing DeleteTablet for tablet e9643e74729946b4afbe133f4d71cb36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:37:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33156
I20260619 14:37:35.020527 2556 tablet_service.cc:1514] Processing DeleteTablet for tablet e9643e74729946b4afbe133f4d71cb36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:37:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45340
I20260619 14:37:35.020926 2785 tablet_replica.cc:330] T e9643e74729946b4afbe133f4d71cb36 P 02b0bac5d7944fb982708b589db96867: stopping tablet replica
I20260619 14:37:35.020980 2783 tablet_replica.cc:330] T e9643e74729946b4afbe133f4d71cb36 P 74f0ef49736744ceaccad487bd25d1cf: stopping tablet replica
I20260619 14:37:35.021008 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 2211
I20260619 14:37:35.021574 2785 raft_consensus.cc:2229] T e9643e74729946b4afbe133f4d71cb36 P 02b0bac5d7944fb982708b589db96867 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:37:35.021750 2785 raft_consensus.cc:2258] T e9643e74729946b4afbe133f4d71cb36 P 02b0bac5d7944fb982708b589db96867 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:37:35.022354 2785 ts_tablet_manager.cc:1905] T e9643e74729946b4afbe133f4d71cb36 P 02b0bac5d7944fb982708b589db96867: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:37:35.023139 2784 tablet_replica.cc:330] T e9643e74729946b4afbe133f4d71cb36 P ff28141737b8451a813f4524e704bbe7: stopping tablet replica
I20260619 14:37:35.023270 2784 raft_consensus.cc:2229] T e9643e74729946b4afbe133f4d71cb36 P ff28141737b8451a813f4524e704bbe7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:37:35.023375 2784 raft_consensus.cc:2258] T e9643e74729946b4afbe133f4d71cb36 P ff28141737b8451a813f4524e704bbe7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:37:35.023967 2784 ts_tablet_manager.cc:1905] T e9643e74729946b4afbe133f4d71cb36 P ff28141737b8451a813f4524e704bbe7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:37:35.024588 2785 ts_tablet_manager.cc:1918] T e9643e74729946b4afbe133f4d71cb36 P 02b0bac5d7944fb982708b589db96867: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 14:37:35.024660 2785 log.cc:1196] T e9643e74729946b4afbe133f4d71cb36 P 02b0bac5d7944fb982708b589db96867: Deleting WAL directory at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/ts-2/wal/wals/e9643e74729946b4afbe133f4d71cb36
I20260619 14:37:35.024905 2785 ts_tablet_manager.cc:1939] T e9643e74729946b4afbe133f4d71cb36 P 02b0bac5d7944fb982708b589db96867: Deleting consensus metadata
I20260619 14:37:35.025458 2125 catalog_manager.cc:4841] TS 02b0bac5d7944fb982708b589db96867 (127.0.98.3:33147): tablet e9643e74729946b4afbe133f4d71cb36 (table test-table [id=5bfb8f4a10bb4e879f1121322708379f]) successfully deleted
I20260619 14:37:35.026261 2784 ts_tablet_manager.cc:1918] T e9643e74729946b4afbe133f4d71cb36 P ff28141737b8451a813f4524e704bbe7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 14:37:35.026333 2784 log.cc:1196] T e9643e74729946b4afbe133f4d71cb36 P ff28141737b8451a813f4524e704bbe7: Deleting WAL directory at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781879837648036-392-0/minicluster-data/ts-1/wal/wals/e9643e74729946b4afbe133f4d71cb36
I20260619 14:37:35.026567 2784 ts_tablet_manager.cc:1939] T e9643e74729946b4afbe133f4d71cb36 P ff28141737b8451a813f4524e704bbe7: Deleting consensus metadata
I20260619 14:37:35.027122 2123 catalog_manager.cc:4841] TS ff28141737b8451a813f4524e704bbe7 (127.0.98.2:42505): tablet e9643e74729946b4afbe133f4d71cb36 (table test-table [id=5bfb8f4a10bb4e879f1121322708379f]) successfully deleted
W20260619 14:37:35.028561 2126 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33225 (error 108)
I20260619 14:37:35.029225 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 2351
I20260619 14:37:35.029933 2787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:35.029101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33225 (local address 127.0.0.1:53840)
0619 14:37:35.029305 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:35.029844 (+ 539us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":64,"threads_started":1}
W20260619 14:37:35.030064 2126 catalog_manager.cc:4584] TS 74f0ef49736744ceaccad487bd25d1cf (127.0.98.1:33225): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e9643e74729946b4afbe133f4d71cb36: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33225: connect: Connection refused (error 111)
I20260619 14:37:35.034219 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 2491
I20260619 14:37:35.039060 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 2107
2026-06-19T14:37:35Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6317 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:37:35 dist-test-slave-32hj krb5kdc[2790](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:37:35 dist-test-slave-32hj krb5kdc[2790](info): set up 2 sockets
Jun 19 14:37:35 dist-test-slave-32hj krb5kdc[2790](info): commencing operation
krb5kdc: starting...
W20260619 14:37:35.992311 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/krb5kdc/test-user.keytab.
Jun 19 14:37:36 dist-test-slave-32hj krb5kdc[2790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:37:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:37:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:37:36.101073 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42293
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42293
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:37:36.205730 2803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:37:36.205973 2803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:37:36.206010 2803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:37:36.209872 2803 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:37:36.209918 2803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:37:36.209935 2803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:37:36.209950 2803 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:37:36.209965 2803 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:37:36.214987 2803 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:37:36.215866 2803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:37:36.216791 2803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:37:36.224509 2808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:36.224599 2809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:36.224525 2811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:36.224735 2803 server_base.cc:1034] running on GCE node
I20260619 14:37:36.225227 2803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:37:36.226210 2803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:37:36.227402 2803 hybrid_clock.cc:648] HybridClock initialized: now 1781879856227375 us; error 40 us; skew 500 ppm
Jun 19 14:37:36 dist-test-slave-32hj krb5kdc[2790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879856, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:37:36.230129 2803 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:37:36.231024 2803 webserver.cc:460] Webserver started at http://127.0.98.62:34801/ using document root <none> and password file <none>
I20260619 14:37:36.231547 2803 fs_manager.cc:359] Metadata directory not provided
I20260619 14:37:36.231614 2803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:37:36.231810 2803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:37:36.233826 2803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/master-0/data/instance:
uuid: "c39a6efb927e41df87e399f4d41e8a9d"
format_stamp: "Formatted at 2026-06-19 14:37:36 on dist-test-slave-32hj"
server_key: "e5ae67b55faefea56d684b71cdc7d237"
server_key_iv: "c54a0286fbed59d1f074c93764a3f790"
server_key_version: "encryptionkey@0"
I20260619 14:37:36.234365 2803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/master-0/wal/instance:
uuid: "c39a6efb927e41df87e399f4d41e8a9d"
format_stamp: "Formatted at 2026-06-19 14:37:36 on dist-test-slave-32hj"
server_key: "e5ae67b55faefea56d684b71cdc7d237"
server_key_iv: "c54a0286fbed59d1f074c93764a3f790"
server_key_version: "encryptionkey@0"
I20260619 14:37:36.237879 2803 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:37:36.240046 2818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:36.241418 2803 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:37:36.241526 2803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/master-0/wal
uuid: "c39a6efb927e41df87e399f4d41e8a9d"
format_stamp: "Formatted at 2026-06-19 14:37:36 on dist-test-slave-32hj"
server_key: "e5ae67b55faefea56d684b71cdc7d237"
server_key_iv: "c54a0286fbed59d1f074c93764a3f790"
server_key_version: "encryptionkey@0"
I20260619 14:37:36.241618 2803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:37:36.265933 2803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:37:36.269019 2803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:37:36.269219 2803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:37:36.275898 2803 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42293
I20260619 14:37:36.275939 2870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42293 every 8 connection(s)
I20260619 14:37:36.276854 2803 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:37:36.279528 2871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:36.285204 2871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c39a6efb927e41df87e399f4d41e8a9d: Bootstrap starting.
I20260619 14:37:36.286372 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 2803
I20260619 14:37:36.286478 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/master-0/wal/instance
I20260619 14:37:36.286712 392 external_mini_cluster.cc:1428] Setting key cf844d9f7584d48f4742615be7edf81d
I20260619 14:37:36.288000 2871 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c39a6efb927e41df87e399f4d41e8a9d: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:36.288772 2871 log.cc:824] T 00000000000000000000000000000000 P c39a6efb927e41df87e399f4d41e8a9d: Log is configured to *not* fsync() on all Append() calls
I20260619 14:37:36.290890 2871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c39a6efb927e41df87e399f4d41e8a9d: No bootstrap required, opened a new log
I20260619 14:37:36.295228 2871 raft_consensus.cc:348] T 00000000000000000000000000000000 P c39a6efb927e41df87e399f4d41e8a9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c39a6efb927e41df87e399f4d41e8a9d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42293 } }
I20260619 14:37:36.295452 2871 raft_consensus.cc:374] T 00000000000000000000000000000000 P c39a6efb927e41df87e399f4d41e8a9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
Jun 19 14:37:36 dist-test-slave-32hj krb5kdc[2790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:37:36.295512 2871 raft_consensus.cc:729] T 00000000000000000000000000000000 P c39a6efb927e41df87e399f4d41e8a9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c39a6efb927e41df87e399f4d41e8a9d, State: Initialized, Role: FOLLOWER
I20260619 14:37:36.296000 2871 consensus_queue.cc:260] T 00000000000000000000000000000000 P c39a6efb927e41df87e399f4d41e8a9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c39a6efb927e41df87e399f4d41e8a9d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42293 } }
I20260619 14:37:36.296166 2871 raft_consensus.cc:388] T 00000000000000000000000000000000 P c39a6efb927e41df87e399f4d41e8a9d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:37:36.296245 2871 raft_consensus.cc:482] T 00000000000000000000000000000000 P c39a6efb927e41df87e399f4d41e8a9d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:37:36.296348 2871 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c39a6efb927e41df87e399f4d41e8a9d [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:36.297255 2871 raft_consensus.cc:504] T 00000000000000000000000000000000 P c39a6efb927e41df87e399f4d41e8a9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c39a6efb927e41df87e399f4d41e8a9d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42293 } }
I20260619 14:37:36.297564 2871 leader_election.cc:302] T 00000000000000000000000000000000 P c39a6efb927e41df87e399f4d41e8a9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c39a6efb927e41df87e399f4d41e8a9d; no voters:
I20260619 14:37:36.297837 2871 leader_election.cc:288] T 00000000000000000000000000000000 P c39a6efb927e41df87e399f4d41e8a9d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:37:36.297930 2876 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c39a6efb927e41df87e399f4d41e8a9d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:37:36.298200 2876 raft_consensus.cc:686] T 00000000000000000000000000000000 P c39a6efb927e41df87e399f4d41e8a9d [term 1 LEADER]: Becoming Leader. State: Replica: c39a6efb927e41df87e399f4d41e8a9d, State: Running, Role: LEADER
I20260619 14:37:36.298527 2876 consensus_queue.cc:237] T 00000000000000000000000000000000 P c39a6efb927e41df87e399f4d41e8a9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c39a6efb927e41df87e399f4d41e8a9d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42293 } }
I20260619 14:37:36.299091 2871 sys_catalog.cc:564] T 00000000000000000000000000000000 P c39a6efb927e41df87e399f4d41e8a9d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:37:36.300063 2877 sys_catalog.cc:455] T 00000000000000000000000000000000 P c39a6efb927e41df87e399f4d41e8a9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c39a6efb927e41df87e399f4d41e8a9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c39a6efb927e41df87e399f4d41e8a9d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42293 } } }
I20260619 14:37:36.300189 2877 sys_catalog.cc:458] T 00000000000000000000000000000000 P c39a6efb927e41df87e399f4d41e8a9d [sys.catalog]: This master's current role is: LEADER
I20260619 14:37:36.300601 2878 sys_catalog.cc:455] T 00000000000000000000000000000000 P c39a6efb927e41df87e399f4d41e8a9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c39a6efb927e41df87e399f4d41e8a9d. Latest consensus state: current_term: 1 leader_uuid: "c39a6efb927e41df87e399f4d41e8a9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c39a6efb927e41df87e399f4d41e8a9d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42293 } } }
I20260619 14:37:36.300766 2878 sys_catalog.cc:458] T 00000000000000000000000000000000 P c39a6efb927e41df87e399f4d41e8a9d [sys.catalog]: This master's current role is: LEADER
I20260619 14:37:36.301481 2874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:36.287793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58718 (local address 127.0.98.62:42293)
0619 14:37:36.288239 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:36.288258 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:36.288293 (+ 35us) server_negotiation.cc:407] Connection header received
0619 14:37:36.288987 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:36.289008 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:36.289343 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:36.289635 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:36.290523 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:36.291553 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:36.292785 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:36.293253 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:36.293369 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:36.296132 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:36.296137 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:36.296152 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:36.296182 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:36.297857 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:36.298388 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:36.298393 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:36.298399 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:36.298532 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:36.298852 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:36.298856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:36.298857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:36.299357 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:36.299548 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:36.300148 (+ 600us) server_negotiation.cc:299] Negotiation successful
0619 14:37:36.300463 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":148,"threads_started":1}
W20260619 14:37:36.303788 2892 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c39a6efb927e41df87e399f4d41e8a9d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:37:36.303912 2892 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:37:36.304003 2885 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:37:36.304626 2885 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:37:36.309247 2885 catalog_manager.cc:1347] Generated new cluster ID: ab6610b4500d4cc49f8a5c7bd7ac8bb8
I20260619 14:37:36.309295 2885 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:37:36.318384 2885 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:37:36.319156 2885 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:37:36.327194 2885 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c39a6efb927e41df87e399f4d41e8a9d: Generated new TSK 0
I20260619 14:37:36.327811 2885 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 14:37:36.353014 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42293
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:37:36.457356 2897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:37:36.457612 2897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:37:36.457653 2897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:37:36.461663 2897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:37:36.461720 2897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:37:36.461776 2897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:37:36.466841 2897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:37:36.467818 2897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:37:36.468775 2897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:37:36.477708 2897 server_base.cc:1034] running on GCE node
W20260619 14:37:36.477643 2902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:36.477792 2905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:36.477643 2903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:36.478238 2897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:37:36.478826 2897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:37:36.479979 2897 hybrid_clock.cc:648] HybridClock initialized: now 1781879856479959 us; error 30 us; skew 500 ppm
Jun 19 14:37:36 dist-test-slave-32hj krb5kdc[2790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879856, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:37:36.482609 2897 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:37:36.483448 2897 webserver.cc:460] Webserver started at http://127.0.98.1:35147/ using document root <none> and password file <none>
I20260619 14:37:36.483958 2897 fs_manager.cc:359] Metadata directory not provided
I20260619 14:37:36.484019 2897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:37:36.484222 2897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:37:36.486313 2897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-0/data/instance:
uuid: "853328bc0ec14842924446df23342ad6"
format_stamp: "Formatted at 2026-06-19 14:37:36 on dist-test-slave-32hj"
server_key: "0beb6a5dfdee8d80a46f67b170a2d31c"
server_key_iv: "62c96a1072b0c3439de1cfd5bb026f85"
server_key_version: "encryptionkey@0"
I20260619 14:37:36.486840 2897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-0/wal/instance:
uuid: "853328bc0ec14842924446df23342ad6"
format_stamp: "Formatted at 2026-06-19 14:37:36 on dist-test-slave-32hj"
server_key: "0beb6a5dfdee8d80a46f67b170a2d31c"
server_key_iv: "62c96a1072b0c3439de1cfd5bb026f85"
server_key_version: "encryptionkey@0"
I20260619 14:37:36.490459 2897 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:37:36.492822 2912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:36.493986 2897 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:37:36.494117 2897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-0/wal
uuid: "853328bc0ec14842924446df23342ad6"
format_stamp: "Formatted at 2026-06-19 14:37:36 on dist-test-slave-32hj"
server_key: "0beb6a5dfdee8d80a46f67b170a2d31c"
server_key_iv: "62c96a1072b0c3439de1cfd5bb026f85"
server_key_version: "encryptionkey@0"
I20260619 14:37:36.494206 2897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:37:36.517675 2897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:37:36.520468 2897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:37:36.520663 2897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:37:36.521327 2897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:37:36.522090 2897 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:37:36.522135 2897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:36.522171 2897 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:37:36.522185 2897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:36.530251 2897 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38235
I20260619 14:37:36.530279 3025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38235 every 8 connection(s)
I20260619 14:37:36.531092 2897 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:37:36.538204 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 2897
I20260619 14:37:36.538307 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:37:36.538558 392 external_mini_cluster.cc:1428] Setting key 21c14077d7c4a7aa8e454d9b5a88f936
Jun 19 14:37:36 dist-test-slave-32hj krb5kdc[2790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879856, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:37:36.543875 2874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:36.532822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53387 (local address 127.0.98.62:42293)
0619 14:37:36.532949 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:36.532952 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:36.533746 (+ 794us) server_negotiation.cc:407] Connection header received
0619 14:37:36.535059 (+ 1313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:36.535063 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:36.535117 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:36.535195 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:36.536333 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:36.536901 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:36.538009 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:36.538254 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:36.538343 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:36.540712 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:36.540715 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:36.540742 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:36.540768 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:36.542032 (+ 1264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:36.542486 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:36.542490 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:36.542491 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:36.542563 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:36.542863 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:36.542867 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:36.542869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:36.543033 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:36.543114 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:36.543597 (+ 483us) server_negotiation.cc:299] Negotiation successful
0619 14:37:36.543716 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 14:37:36.544694 3028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:36.533184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42293 (local address 127.0.98.1:53387)
0619 14:37:36.533606 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:36.533648 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:36.534856 (+ 1208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:36.535367 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:36.535374 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:36.535716 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:36.536191 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:36.536206 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:36.537024 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:36.537026 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:36.537885 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:36.537894 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:36.538393 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:36.538397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:36.538552 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:36.539080 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:36.539103 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:36.540562 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:36.542140 (+ 1578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:36.542146 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:36.542161 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:36.542398 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:36.542649 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:36.542652 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:36.542654 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:36.542782 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:36.543134 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:36.543140 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:36.543373 (+ 233us) client_negotiation.cc:769] Sending connection context
0619 14:37:36.543535 (+ 162us) client_negotiation.cc:240] Negotiation successful
0619 14:37:36.543845 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":111,"threads_started":1}
I20260619 14:37:36.545850 3026 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42293
I20260619 14:37:36.546066 3026 heartbeater.cc:464] Registering TS with master...
I20260619 14:37:36.546514 3026 heartbeater.cc:511] Master 127.0.98.62:42293 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 14:37:36.547892 2834 ts_manager.cc:194] Registered new tserver with Master: 853328bc0ec14842924446df23342ad6 (127.0.98.1:38235)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 14:37:36.549177 2834 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53387
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 14:37:36.561619 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42293
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:37:36.663872 3031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:37:36.664093 3031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:37:36.664137 3031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:37:36.668030 3031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:37:36.668077 3031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:37:36.668148 3031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:37:36.673064 3031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:37:36.673930 3031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:37:36.674844 3031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:37:36.683457 3036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:36.683434 3039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:36.683441 3037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:36.683620 3031 server_base.cc:1034] running on GCE node
I20260619 14:37:36.684134 3031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:37:36.684703 3031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:37:36.685868 3031 hybrid_clock.cc:648] HybridClock initialized: now 1781879856685847 us; error 35 us; skew 500 ppm
Jun 19 14:37:36 dist-test-slave-32hj krb5kdc[2790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879856, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:37:36.688369 3031 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:37:36.689170 3031 webserver.cc:460] Webserver started at http://127.0.98.2:36877/ using document root <none> and password file <none>
I20260619 14:37:36.689668 3031 fs_manager.cc:359] Metadata directory not provided
I20260619 14:37:36.689725 3031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:37:36.689925 3031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:37:36.691824 3031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-1/data/instance:
uuid: "86b0623199e14bad99b2544d924602be"
format_stamp: "Formatted at 2026-06-19 14:37:36 on dist-test-slave-32hj"
server_key: "c827f5800f1a36cc219b032b4c922d6f"
server_key_iv: "4c30971b12d4fcef08a8d0d7bf6a95ce"
server_key_version: "encryptionkey@0"
I20260619 14:37:36.692327 3031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-1/wal/instance:
uuid: "86b0623199e14bad99b2544d924602be"
format_stamp: "Formatted at 2026-06-19 14:37:36 on dist-test-slave-32hj"
server_key: "c827f5800f1a36cc219b032b4c922d6f"
server_key_iv: "4c30971b12d4fcef08a8d0d7bf6a95ce"
server_key_version: "encryptionkey@0"
I20260619 14:37:36.695721 3031 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.001s
I20260619 14:37:36.697944 3046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:36.699084 3031 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:37:36.699185 3031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-1/wal
uuid: "86b0623199e14bad99b2544d924602be"
format_stamp: "Formatted at 2026-06-19 14:37:36 on dist-test-slave-32hj"
server_key: "c827f5800f1a36cc219b032b4c922d6f"
server_key_iv: "4c30971b12d4fcef08a8d0d7bf6a95ce"
server_key_version: "encryptionkey@0"
I20260619 14:37:36.699266 3031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:37:36.708920 3031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:37:36.711207 3031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:37:36.711352 3031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:37:36.711840 3031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:37:36.712541 3031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:37:36.712585 3031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:36.712651 3031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:37:36.712677 3031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:36.720409 3031 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35603
I20260619 14:37:36.720460 3159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35603 every 8 connection(s)
I20260619 14:37:36.721276 3031 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:37:36.726485 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 3031
I20260619 14:37:36.726584 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:37:36.726826 392 external_mini_cluster.cc:1428] Setting key e20ddfaa25301ce60bb1290166b80745
Jun 19 14:37:36 dist-test-slave-32hj krb5kdc[2790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879856, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:37:36.734843 2874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:36.722953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53017 (local address 127.0.98.62:42293)
0619 14:37:36.723085 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:36.723088 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:36.723940 (+ 852us) server_negotiation.cc:407] Connection header received
0619 14:37:36.725314 (+ 1374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:36.725317 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:36.725372 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:36.725443 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:36.726583 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:36.727150 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:36.728530 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:36.728786 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:36.728869 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:36.731485 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:36.731487 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:36.731489 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:36.731513 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:36.732806 (+ 1293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:36.733381 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:36.733384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:36.733385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:36.733463 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:36.733817 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:36.733818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:36.733820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:36.733988 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:36.734056 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:36.734634 (+ 578us) server_negotiation.cc:299] Negotiation successful
0619 14:37:36.734723 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 14:37:36.735978 3162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:36.723367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42293 (local address 127.0.98.2:53017)
0619 14:37:36.723798 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:36.723840 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:36.725116 (+ 1276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:36.725614 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:36.725622 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:36.725956 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:36.726452 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:36.726466 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:36.727272 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:36.727275 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:36.728420 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:36.728431 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:36.728944 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:36.728948 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:36.729141 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:36.729713 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:36.729729 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:36.731334 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:36.732908 (+ 1574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:36.732913 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:36.732928 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:36.733277 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:36.733546 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:36.733550 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:36.733552 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:36.733735 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:36.734078 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:36.734084 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:36.734392 (+ 308us) client_negotiation.cc:769] Sending connection context
0619 14:37:36.734610 (+ 218us) client_negotiation.cc:240] Negotiation successful
0619 14:37:36.734933 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":115,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 14:37:36.736997 3160 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42293
I20260619 14:37:36.737229 3160 heartbeater.cc:464] Registering TS with master...
I20260619 14:37:36.737670 3160 heartbeater.cc:511] Master 127.0.98.62:42293 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 14:37:36.738651 2834 ts_manager.cc:194] Registered new tserver with Master: 86b0623199e14bad99b2544d924602be (127.0.98.2:35603)
I20260619 14:37:36.739182 2834 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53017
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 14:37:36.751168 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42293
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:37:36.856606 3165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:37:36.856860 3165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:37:36.856901 3165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:37:36.860792 3165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:37:36.860845 3165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:37:36.860908 3165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:37:36.865897 3165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:37:36.866837 3165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:37:36.867784 3165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:37:36.876679 3173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:36.876842 3165 server_base.cc:1034] running on GCE node
W20260619 14:37:36.876827 3170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:36.876830 3171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:36.877422 3165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:37:36.877997 3165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:37:36.879179 3165 hybrid_clock.cc:648] HybridClock initialized: now 1781879856879152 us; error 34 us; skew 500 ppm
Jun 19 14:37:36 dist-test-slave-32hj krb5kdc[2790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879856, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:37:36.881840 3165 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:37:36.882728 3165 webserver.cc:460] Webserver started at http://127.0.98.3:46117/ using document root <none> and password file <none>
I20260619 14:37:36.883430 3165 fs_manager.cc:359] Metadata directory not provided
I20260619 14:37:36.883512 3165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:37:36.883699 3165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:37:36.885680 3165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-2/data/instance:
uuid: "8bf253f5f3e44a80beb9229278df82c9"
format_stamp: "Formatted at 2026-06-19 14:37:36 on dist-test-slave-32hj"
server_key: "810925b8e88fd32e102581ae9cc6d5a1"
server_key_iv: "62b4ca713add16fa2461ce521e41d9c5"
server_key_version: "encryptionkey@0"
I20260619 14:37:36.886183 3165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8bf253f5f3e44a80beb9229278df82c9"
format_stamp: "Formatted at 2026-06-19 14:37:36 on dist-test-slave-32hj"
server_key: "810925b8e88fd32e102581ae9cc6d5a1"
server_key_iv: "62b4ca713add16fa2461ce521e41d9c5"
server_key_version: "encryptionkey@0"
I20260619 14:37:36.889520 3165 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:37:36.891711 3180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:36.892874 3165 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:37:36.892987 3165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-2/wal
uuid: "8bf253f5f3e44a80beb9229278df82c9"
format_stamp: "Formatted at 2026-06-19 14:37:36 on dist-test-slave-32hj"
server_key: "810925b8e88fd32e102581ae9cc6d5a1"
server_key_iv: "62b4ca713add16fa2461ce521e41d9c5"
server_key_version: "encryptionkey@0"
I20260619 14:37:36.893072 3165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:37:36.917482 3165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:37:36.920444 3165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:37:36.920598 3165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:37:36.921177 3165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:37:36.921936 3165 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:37:36.921978 3165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:36.922021 3165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:37:36.922048 3165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:36.930032 3165 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35511
I20260619 14:37:36.930078 3293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35511 every 8 connection(s)
I20260619 14:37:36.930830 3165 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:37:36.936836 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 3165
I20260619 14:37:36.936949 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:37:36.937249 392 external_mini_cluster.cc:1428] Setting key ab230f92c2a5f9043a0fab84b6ecff8b
Jun 19 14:37:36 dist-test-slave-32hj krb5kdc[2790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879856, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:37:36.943708 2874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:36.932429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40447 (local address 127.0.98.62:42293)
0619 14:37:36.932558 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:36.932561 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:36.933302 (+ 741us) server_negotiation.cc:407] Connection header received
0619 14:37:36.934685 (+ 1383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:36.934688 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:36.934744 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:36.934817 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:36.935922 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:36.936440 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:36.937874 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:36.938162 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:36.938243 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:36.940571 (+ 2328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:36.940574 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:36.940576 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:36.940601 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:36.941868 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:36.942346 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:36.942350 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:36.942351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:36.942438 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:36.942759 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:36.942762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:36.942763 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:36.942957 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:36.943038 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:36.943492 (+ 454us) server_negotiation.cc:299] Negotiation successful
0619 14:37:36.943593 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 14:37:36.944487 3296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:36.932750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42293 (local address 127.0.98.3:40447)
0619 14:37:36.933160 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:36.933201 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:36.934439 (+ 1238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:36.934978 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:36.934985 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:36.935317 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:36.935777 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:36.935790 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:36.936551 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:36.936554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:36.937774 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:36.937783 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:36.938300 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:36.938304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:36.938460 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:36.938971 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:36.938990 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:36.940404 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:36.941979 (+ 1575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:36.941984 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:36.941996 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:36.942240 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:36.942523 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:36.942526 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:36.942527 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:36.942656 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:36.943035 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:36.943040 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:36.943260 (+ 220us) client_negotiation.cc:769] Sending connection context
0619 14:37:36.943417 (+ 157us) client_negotiation.cc:240] Negotiation successful
0619 14:37:36.943662 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":101,"threads_started":1}
I20260619 14:37:36.945658 3294 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42293
I20260619 14:37:36.945879 3294 heartbeater.cc:464] Registering TS with master...
I20260619 14:37:36.946316 3294 heartbeater.cc:511] Master 127.0.98.62:42293 requested a full tablet report, sending...
I20260619 14:37:36.947243 2834 ts_manager.cc:194] Registered new tserver with Master: 8bf253f5f3e44a80beb9229278df82c9 (127.0.98.3:35511)
I20260619 14:37:36.947752 2834 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40447
I20260619 14:37:36.951057 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 14:37:36.962605 2874 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:37:36.961177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58730 (local address 127.0.98.62:42293)
0619 14:37:36.961305 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:36.961309 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:36.961482 (+ 173us) server_negotiation.cc:407] Connection header received
0619 14:37:36.961635 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:36.961638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:36.961681 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:36.961747 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:36.962468 (+ 721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58730: BlockingRecv error: recv got EOF from 127.0.0.1:58730 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 14:37:36.963793 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 2897
I20260619 14:37:36.968556 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 3031
I20260619 14:37:36.972991 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 3165
I20260619 14:37:36.977272 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 2803
2026-06-19T14:37:36Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1935 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781879837648036-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:37:37 dist-test-slave-32hj krb5kdc[3306](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:37:37 dist-test-slave-32hj krb5kdc[3306](info): set up 2 sockets
Jun 19 14:37:37 dist-test-slave-32hj krb5kdc[3306](info): commencing operation
krb5kdc: starting...
W20260619 14:37:37.928592 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781879837648036-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781879837648036-392-0/krb5kdc/test-user.keytab.
Jun 19 14:37:38 dist-test-slave-32hj krb5kdc[3306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:37:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:37:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:37:38.036700 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781879837648036-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781879837648036-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781879837648036-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42291
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42291
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:37:38.141472 3319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:37:38.141731 3319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:37:38.141767 3319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:37:38.145674 3319 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:37:38.145725 3319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:37:38.145743 3319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:37:38.145757 3319 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:37:38.145771 3319 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:37:38.150774 3319 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781879837648036-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781879837648036-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42291
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781879837648036-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:37:38.151758 3319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:37:38.152755 3319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:37:38.160413 3324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:38.160442 3319 server_base.cc:1034] running on GCE node
W20260619 14:37:38.160537 3327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:38.160447 3325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:38.161139 3319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:37:38.162142 3319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:37:38.163290 3319 hybrid_clock.cc:648] HybridClock initialized: now 1781879858163265 us; error 36 us; skew 500 ppm
Jun 19 14:37:38 dist-test-slave-32hj krb5kdc[3306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879858, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:37:38.165824 3319 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:37:38.166620 3319 webserver.cc:460] Webserver started at http://127.0.98.62:42687/ using document root <none> and password file <none>
I20260619 14:37:38.167105 3319 fs_manager.cc:359] Metadata directory not provided
I20260619 14:37:38.167167 3319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:37:38.167361 3319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:37:38.169284 3319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781879837648036-392-0/minicluster-data/master-0/data/instance:
uuid: "d5a34d20b6164f8da1cb9656f1b6fc38"
format_stamp: "Formatted at 2026-06-19 14:37:38 on dist-test-slave-32hj"
server_key: "360e7ce6a6f01de17d1ee29c346aa84f"
server_key_iv: "f67432e6e0ab200b3cf27eca7e77456c"
server_key_version: "encryptionkey@0"
I20260619 14:37:38.169835 3319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781879837648036-392-0/minicluster-data/master-0/wal/instance:
uuid: "d5a34d20b6164f8da1cb9656f1b6fc38"
format_stamp: "Formatted at 2026-06-19 14:37:38 on dist-test-slave-32hj"
server_key: "360e7ce6a6f01de17d1ee29c346aa84f"
server_key_iv: "f67432e6e0ab200b3cf27eca7e77456c"
server_key_version: "encryptionkey@0"
I20260619 14:37:38.173229 3319 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:37:38.175304 3334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:38.176397 3319 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:37:38.176502 3319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781879837648036-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781879837648036-392-0/minicluster-data/master-0/wal
uuid: "d5a34d20b6164f8da1cb9656f1b6fc38"
format_stamp: "Formatted at 2026-06-19 14:37:38 on dist-test-slave-32hj"
server_key: "360e7ce6a6f01de17d1ee29c346aa84f"
server_key_iv: "f67432e6e0ab200b3cf27eca7e77456c"
server_key_version: "encryptionkey@0"
I20260619 14:37:38.176597 3319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781879837648036-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781879837648036-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781879837648036-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:37:38.194321 3319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:37:38.197333 3319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:37:38.197497 3319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:37:38.204174 3319 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42291
I20260619 14:37:38.204413 3386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42291 every 8 connection(s)
I20260619 14:37:38.205120 3319 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:37:38.207697 3387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:38.211890 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 3319
I20260619 14:37:38.211982 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781879837648036-392-0/minicluster-data/master-0/wal/instance
I20260619 14:37:38.212231 392 external_mini_cluster.cc:1428] Setting key 1c2456cc8cda37cb5734c8b61e408265
I20260619 14:37:38.213029 3387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5a34d20b6164f8da1cb9656f1b6fc38: Bootstrap starting.
I20260619 14:37:38.215515 3387 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5a34d20b6164f8da1cb9656f1b6fc38: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:38.216307 3387 log.cc:824] T 00000000000000000000000000000000 P d5a34d20b6164f8da1cb9656f1b6fc38: Log is configured to *not* fsync() on all Append() calls
I20260619 14:37:38.218189 3387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5a34d20b6164f8da1cb9656f1b6fc38: No bootstrap required, opened a new log
Jun 19 14:37:38 dist-test-slave-32hj krb5kdc[3306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:37:38.221693 3387 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5a34d20b6164f8da1cb9656f1b6fc38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5a34d20b6164f8da1cb9656f1b6fc38" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42291 } }
I20260619 14:37:38.221956 3387 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5a34d20b6164f8da1cb9656f1b6fc38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:38.222012 3387 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5a34d20b6164f8da1cb9656f1b6fc38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5a34d20b6164f8da1cb9656f1b6fc38, State: Initialized, Role: FOLLOWER
I20260619 14:37:38.222524 3387 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5a34d20b6164f8da1cb9656f1b6fc38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5a34d20b6164f8da1cb9656f1b6fc38" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42291 } }
I20260619 14:37:38.222678 3387 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5a34d20b6164f8da1cb9656f1b6fc38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:37:38.222740 3387 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5a34d20b6164f8da1cb9656f1b6fc38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:37:38.222826 3387 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5a34d20b6164f8da1cb9656f1b6fc38 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:38.223740 3387 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5a34d20b6164f8da1cb9656f1b6fc38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5a34d20b6164f8da1cb9656f1b6fc38" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42291 } }
I20260619 14:37:38.224053 3387 leader_election.cc:302] T 00000000000000000000000000000000 P d5a34d20b6164f8da1cb9656f1b6fc38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5a34d20b6164f8da1cb9656f1b6fc38; no voters:
I20260619 14:37:38.224327 3387 leader_election.cc:288] T 00000000000000000000000000000000 P d5a34d20b6164f8da1cb9656f1b6fc38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:37:38.224402 3392 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5a34d20b6164f8da1cb9656f1b6fc38 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:37:38.224599 3392 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5a34d20b6164f8da1cb9656f1b6fc38 [term 1 LEADER]: Becoming Leader. State: Replica: d5a34d20b6164f8da1cb9656f1b6fc38, State: Running, Role: LEADER
I20260619 14:37:38.224958 3392 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5a34d20b6164f8da1cb9656f1b6fc38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5a34d20b6164f8da1cb9656f1b6fc38" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42291 } }
I20260619 14:37:38.225531 3387 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5a34d20b6164f8da1cb9656f1b6fc38 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:37:38.226408 3390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:38.213494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42686 (local address 127.0.98.62:42291)
0619 14:37:38.213880 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:38.213897 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:38.213937 (+ 40us) server_negotiation.cc:407] Connection header received
0619 14:37:38.214745 (+ 808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:38.214767 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:38.215162 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:38.215500 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:38.216184 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:38.217506 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:38.218312 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:38.218963 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:38.219117 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:38.221261 (+ 2144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:38.221267 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:38.221282 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:38.221311 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:38.223184 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:38.223553 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:38.223558 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:38.223564 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:38.223711 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:38.223953 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:38.223956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:38.223957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:38.224376 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:38.225217 (+ 841us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:38.225330 (+ 113us) server_negotiation.cc:299] Negotiation successful
0619 14:37:38.225545 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":89,"threads_started":1}
I20260619 14:37:38.226621 3394 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5a34d20b6164f8da1cb9656f1b6fc38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5a34d20b6164f8da1cb9656f1b6fc38. Latest consensus state: current_term: 1 leader_uuid: "d5a34d20b6164f8da1cb9656f1b6fc38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5a34d20b6164f8da1cb9656f1b6fc38" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42291 } } }
I20260619 14:37:38.226660 3393 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5a34d20b6164f8da1cb9656f1b6fc38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5a34d20b6164f8da1cb9656f1b6fc38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5a34d20b6164f8da1cb9656f1b6fc38" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42291 } } }
I20260619 14:37:38.226725 3394 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5a34d20b6164f8da1cb9656f1b6fc38 [sys.catalog]: This master's current role is: LEADER
I20260619 14:37:38.226749 3393 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5a34d20b6164f8da1cb9656f1b6fc38 [sys.catalog]: This master's current role is: LEADER
I20260619 14:37:38.227088 3401 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:37:38.229580 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:37:38.230188 3401 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:37:38.235150 3401 catalog_manager.cc:1347] Generated new cluster ID: caacb6b8739946e686e6f0d420ff83e5
I20260619 14:37:38.235210 3401 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:37:38.249959 3401 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:37:38.251329 3401 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:37:38.256573 3390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:38.251013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42700 (local address 127.0.98.62:42291)
0619 14:37:38.251121 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:38.251126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:38.251423 (+ 297us) server_negotiation.cc:407] Connection header received
0619 14:37:38.251586 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:38.251589 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:38.251640 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:38.251716 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:38.252282 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:38.253169 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:38.254313 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:38.254680 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:38.254780 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:38.255602 (+ 822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:38.255605 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:38.255608 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:37:38.256158 (+ 550us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:38.256190 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:37:38.256210 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:38.256290 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:38.256362 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 14:37:38.256421 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":40}
I20260619 14:37:38.260013 3401 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5a34d20b6164f8da1cb9656f1b6fc38: Generated new TSK 0
I20260619 14:37:38.260763 3401 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:37:38.267995 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 3319
2026-06-19T14:37:38Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1289 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:37:38 dist-test-slave-32hj krb5kdc[3420](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:37:38 dist-test-slave-32hj krb5kdc[3420](info): set up 2 sockets
Jun 19 14:37:38 dist-test-slave-32hj krb5kdc[3420](info): commencing operation
krb5kdc: starting...
W20260619 14:37:39.224406 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/krb5kdc/test-user.keytab.
Jun 19 14:37:39 dist-test-slave-32hj krb5kdc[3420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:37:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:37:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:37:39.329742 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40527
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40527
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:37:39.431241 3433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:37:39.431459 3433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:37:39.431497 3433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:37:39.435293 3433 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:37:39.435338 3433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:37:39.435355 3433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:37:39.435369 3433 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:37:39.435382 3433 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:37:39.440233 3433 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40527
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:37:39.441092 3433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:37:39.441933 3433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:37:39.449509 3439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:39.449536 3441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:39.449536 3438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:39.449874 3433 server_base.cc:1034] running on GCE node
I20260619 14:37:39.450387 3433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:37:39.451437 3433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:37:39.452631 3433 hybrid_clock.cc:648] HybridClock initialized: now 1781879859452612 us; error 35 us; skew 500 ppm
Jun 19 14:37:39 dist-test-slave-32hj krb5kdc[3420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879859, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:37:39.455312 3433 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:37:39.456163 3433 webserver.cc:460] Webserver started at http://127.0.98.62:40453/ using document root <none> and password file <none>
I20260619 14:37:39.456668 3433 fs_manager.cc:359] Metadata directory not provided
I20260619 14:37:39.456759 3433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:37:39.456987 3433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:37:39.458940 3433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/master-0/data/instance:
uuid: "c5fa0ef72a8140d9a3ec6c5896d1125a"
format_stamp: "Formatted at 2026-06-19 14:37:39 on dist-test-slave-32hj"
server_key: "fd745cad4c9a67683fc47e4e47ef235b"
server_key_iv: "40aade7315fc4e6841ce79c8a0257a7d"
server_key_version: "encryptionkey@0"
I20260619 14:37:39.459479 3433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/master-0/wal/instance:
uuid: "c5fa0ef72a8140d9a3ec6c5896d1125a"
format_stamp: "Formatted at 2026-06-19 14:37:39 on dist-test-slave-32hj"
server_key: "fd745cad4c9a67683fc47e4e47ef235b"
server_key_iv: "40aade7315fc4e6841ce79c8a0257a7d"
server_key_version: "encryptionkey@0"
I20260619 14:37:39.463071 3433 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260619 14:37:39.465476 3448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:39.466739 3433 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 14:37:39.466861 3433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/master-0/wal
uuid: "c5fa0ef72a8140d9a3ec6c5896d1125a"
format_stamp: "Formatted at 2026-06-19 14:37:39 on dist-test-slave-32hj"
server_key: "fd745cad4c9a67683fc47e4e47ef235b"
server_key_iv: "40aade7315fc4e6841ce79c8a0257a7d"
server_key_version: "encryptionkey@0"
I20260619 14:37:39.466960 3433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:37:39.478868 3433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:37:39.481470 3433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:37:39.481643 3433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:37:39.488082 3433 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40527
I20260619 14:37:39.488322 3500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40527 every 8 connection(s)
I20260619 14:37:39.489030 3433 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:37:39.491679 3501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:39.494725 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 3433
I20260619 14:37:39.494837 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/master-0/wal/instance
I20260619 14:37:39.495153 392 external_mini_cluster.cc:1428] Setting key d75e768766b04d4215ee54646dc50971
I20260619 14:37:39.497490 3501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5fa0ef72a8140d9a3ec6c5896d1125a: Bootstrap starting.
I20260619 14:37:39.499536 3501 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5fa0ef72a8140d9a3ec6c5896d1125a: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:39.500193 3501 log.cc:824] T 00000000000000000000000000000000 P c5fa0ef72a8140d9a3ec6c5896d1125a: Log is configured to *not* fsync() on all Append() calls
I20260619 14:37:39.501895 3501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5fa0ef72a8140d9a3ec6c5896d1125a: No bootstrap required, opened a new log
Jun 19 14:37:39 dist-test-slave-32hj krb5kdc[3420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:37:39.505705 3501 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5fa0ef72a8140d9a3ec6c5896d1125a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5fa0ef72a8140d9a3ec6c5896d1125a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40527 } }
I20260619 14:37:39.505971 3501 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5fa0ef72a8140d9a3ec6c5896d1125a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:39.506037 3501 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5fa0ef72a8140d9a3ec6c5896d1125a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5fa0ef72a8140d9a3ec6c5896d1125a, State: Initialized, Role: FOLLOWER
I20260619 14:37:39.506503 3501 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5fa0ef72a8140d9a3ec6c5896d1125a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5fa0ef72a8140d9a3ec6c5896d1125a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40527 } }
I20260619 14:37:39.506652 3501 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5fa0ef72a8140d9a3ec6c5896d1125a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:37:39.506716 3501 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5fa0ef72a8140d9a3ec6c5896d1125a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:37:39.506806 3501 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5fa0ef72a8140d9a3ec6c5896d1125a [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:39.507752 3501 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5fa0ef72a8140d9a3ec6c5896d1125a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5fa0ef72a8140d9a3ec6c5896d1125a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40527 } }
I20260619 14:37:39.508050 3501 leader_election.cc:302] T 00000000000000000000000000000000 P c5fa0ef72a8140d9a3ec6c5896d1125a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5fa0ef72a8140d9a3ec6c5896d1125a; no voters:
I20260619 14:37:39.508306 3501 leader_election.cc:288] T 00000000000000000000000000000000 P c5fa0ef72a8140d9a3ec6c5896d1125a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:37:39.508358 3504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:39.496483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33422 (local address 127.0.98.62:40527)
0619 14:37:39.496911 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:39.496926 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:39.496960 (+ 34us) server_negotiation.cc:407] Connection header received
0619 14:37:39.497571 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:39.497589 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:39.497919 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:39.498227 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:39.498974 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:39.499982 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:39.500851 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:39.501318 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:39.501467 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:39.504043 (+ 2576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:39.504048 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:39.504060 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:39.504085 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:39.505682 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:39.506158 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:39.506163 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:39.506167 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:39.506273 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:39.506522 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:39.506524 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:39.506525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:39.506929 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:39.507086 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:39.507339 (+ 253us) server_negotiation.cc:299] Negotiation successful
0619 14:37:39.507589 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":152,"threads_started":1}
I20260619 14:37:39.508402 3506 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5fa0ef72a8140d9a3ec6c5896d1125a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:37:39.508584 3506 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5fa0ef72a8140d9a3ec6c5896d1125a [term 1 LEADER]: Becoming Leader. State: Replica: c5fa0ef72a8140d9a3ec6c5896d1125a, State: Running, Role: LEADER
I20260619 14:37:39.508948 3506 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5fa0ef72a8140d9a3ec6c5896d1125a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5fa0ef72a8140d9a3ec6c5896d1125a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40527 } }
I20260619 14:37:39.509447 3501 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5fa0ef72a8140d9a3ec6c5896d1125a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:37:39.511008 3508 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5fa0ef72a8140d9a3ec6c5896d1125a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5fa0ef72a8140d9a3ec6c5896d1125a. Latest consensus state: current_term: 1 leader_uuid: "c5fa0ef72a8140d9a3ec6c5896d1125a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5fa0ef72a8140d9a3ec6c5896d1125a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40527 } } }
I20260619 14:37:39.511132 3508 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5fa0ef72a8140d9a3ec6c5896d1125a [sys.catalog]: This master's current role is: LEADER
I20260619 14:37:39.511406 3507 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5fa0ef72a8140d9a3ec6c5896d1125a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5fa0ef72a8140d9a3ec6c5896d1125a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5fa0ef72a8140d9a3ec6c5896d1125a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40527 } } }
I20260619 14:37:39.511546 3507 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5fa0ef72a8140d9a3ec6c5896d1125a [sys.catalog]: This master's current role is: LEADER
W20260619 14:37:39.513849 3522 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c5fa0ef72a8140d9a3ec6c5896d1125a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:37:39.513919 3522 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:37:39.513971 3520 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:37:39.514652 3520 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:37:39.519191 3520 catalog_manager.cc:1347] Generated new cluster ID: d7931f921be14a19becacb073c19d5bd
I20260619 14:37:39.519253 3520 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:37:39.535550 3520 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:37:39.536446 3520 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:37:39.545395 3520 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5fa0ef72a8140d9a3ec6c5896d1125a: Generated new TSK 0
I20260619 14:37:39.546046 3520 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 14:37:39.580228 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40527
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:37:39.684991 3527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:37:39.685227 3527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:37:39.685267 3527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:37:39.689134 3527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:37:39.689181 3527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:37:39.689236 3527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:37:39.694173 3527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3527
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:37:39.695093 3527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:37:39.695977 3527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:37:39.704638 3532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:39.704649 3533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:39.704746 3527 server_base.cc:1034] running on GCE node
W20260619 14:37:39.704659 3535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:39.705286 3527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:37:39.705844 3527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:37:39.707003 3527 hybrid_clock.cc:648] HybridClock initialized: now 1781879859706977 us; error 38 us; skew 500 ppm
Jun 19 14:37:39 dist-test-slave-32hj krb5kdc[3420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879859, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:37:39.709532 3527 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:37:39.710453 3527 webserver.cc:460] Webserver started at http://127.0.98.1:37607/ using document root <none> and password file <none>
I20260619 14:37:39.711010 3527 fs_manager.cc:359] Metadata directory not provided
I20260619 14:37:39.711081 3527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:37:39.711301 3527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:37:39.713362 3527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-0/data/instance:
uuid: "8ca6d068deb2406ebdb1534843bc691f"
format_stamp: "Formatted at 2026-06-19 14:37:39 on dist-test-slave-32hj"
server_key: "7f99f84877a169121af2dc59465097d7"
server_key_iv: "62606691c6906aa1ab66db21d5b522f0"
server_key_version: "encryptionkey@0"
I20260619 14:37:39.713896 3527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8ca6d068deb2406ebdb1534843bc691f"
format_stamp: "Formatted at 2026-06-19 14:37:39 on dist-test-slave-32hj"
server_key: "7f99f84877a169121af2dc59465097d7"
server_key_iv: "62606691c6906aa1ab66db21d5b522f0"
server_key_version: "encryptionkey@0"
I20260619 14:37:39.717447 3527 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:37:39.719707 3542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:39.720934 3527 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:37:39.721040 3527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-0/wal
uuid: "8ca6d068deb2406ebdb1534843bc691f"
format_stamp: "Formatted at 2026-06-19 14:37:39 on dist-test-slave-32hj"
server_key: "7f99f84877a169121af2dc59465097d7"
server_key_iv: "62606691c6906aa1ab66db21d5b522f0"
server_key_version: "encryptionkey@0"
I20260619 14:37:39.721127 3527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:37:39.736692 3527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:37:39.739246 3527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:37:39.739404 3527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:37:39.739919 3527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:37:39.740661 3527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:37:39.740706 3527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:39.740778 3527 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:37:39.740804 3527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:39.748961 3527 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36491
I20260619 14:37:39.748997 3655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36491 every 8 connection(s)
I20260619 14:37:39.749743 3527 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:37:39.755393 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 3527
I20260619 14:37:39.755506 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:37:39.755786 392 external_mini_cluster.cc:1428] Setting key 55b3d2625d8b433830d8f6736c7abdfd
Jun 19 14:37:39 dist-test-slave-32hj krb5kdc[3420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879859, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:37:39.763749 3504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:39.751318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39465 (local address 127.0.98.62:40527)
0619 14:37:39.751474 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:39.751478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:39.752313 (+ 835us) server_negotiation.cc:407] Connection header received
0619 14:37:39.753708 (+ 1395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:39.753711 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:39.753756 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:39.753828 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:39.755260 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:39.756026 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:39.757345 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:39.757670 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:39.757767 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:39.760225 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:39.760229 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:39.760232 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:39.760258 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:39.761767 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:39.762222 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:39.762226 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:39.762228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:39.762315 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:39.762674 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:39.762677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:39.762678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:39.762885 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:39.762985 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:39.763470 (+ 485us) server_negotiation.cc:299] Negotiation successful
0619 14:37:39.763585 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 14:37:39.764498 3658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:39.751715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40527 (local address 127.0.98.1:39465)
0619 14:37:39.752163 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:39.752206 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:39.753509 (+ 1303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:39.754122 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:39.754129 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:39.754481 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:39.755093 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:39.755111 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:39.756169 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:39.756173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:39.757237 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:39.757246 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:39.757813 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:39.757816 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:39.757970 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:39.758554 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:39.758574 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:39.760077 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:39.761883 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:39.761887 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:39.761898 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:39.762133 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:39.762428 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:39.762431 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:39.762432 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:39.762580 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:39.763008 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:39.763013 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:39.763232 (+ 219us) client_negotiation.cc:769] Sending connection context
0619 14:37:39.763405 (+ 173us) client_negotiation.cc:240] Negotiation successful
0619 14:37:39.763647 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":114,"threads_started":1}
I20260619 14:37:39.765702 3656 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40527
I20260619 14:37:39.765935 3656 heartbeater.cc:464] Registering TS with master...
I20260619 14:37:39.766408 3656 heartbeater.cc:511] Master 127.0.98.62:40527 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 14:37:39.767825 3465 ts_manager.cc:194] Registered new tserver with Master: 8ca6d068deb2406ebdb1534843bc691f (127.0.98.1:36491)
I20260619 14:37:39.768976 3465 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39465
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 14:37:39.779500 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40527
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:37:39.885918 3661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:37:39.886180 3661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:37:39.886240 3661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:37:39.890214 3661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:37:39.890268 3661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:37:39.890343 3661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:37:39.895663 3661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:37:39.896646 3661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:37:39.897635 3661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:37:39.906584 3666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:39.906663 3667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:39.906639 3661 server_base.cc:1034] running on GCE node
W20260619 14:37:39.906734 3669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:39.907132 3661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:37:39.907747 3661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:37:39.908885 3661 hybrid_clock.cc:648] HybridClock initialized: now 1781879859908860 us; error 42 us; skew 500 ppm
Jun 19 14:37:39 dist-test-slave-32hj krb5kdc[3420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879859, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:37:39.911726 3661 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:37:39.912485 3661 webserver.cc:460] Webserver started at http://127.0.98.2:44111/ using document root <none> and password file <none>
I20260619 14:37:39.913041 3661 fs_manager.cc:359] Metadata directory not provided
I20260619 14:37:39.913100 3661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:37:39.913260 3661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:37:39.915118 3661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-1/data/instance:
uuid: "c21c5a2c8f6d4fb184eb93acd9821484"
format_stamp: "Formatted at 2026-06-19 14:37:39 on dist-test-slave-32hj"
server_key: "e881b393d57dbc95b9dd0aa27d645bd5"
server_key_iv: "cdcf3dc9d955d374e660eac43872de7b"
server_key_version: "encryptionkey@0"
I20260619 14:37:39.915597 3661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c21c5a2c8f6d4fb184eb93acd9821484"
format_stamp: "Formatted at 2026-06-19 14:37:39 on dist-test-slave-32hj"
server_key: "e881b393d57dbc95b9dd0aa27d645bd5"
server_key_iv: "cdcf3dc9d955d374e660eac43872de7b"
server_key_version: "encryptionkey@0"
I20260619 14:37:39.919281 3661 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260619 14:37:39.921615 3676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:39.922793 3661 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:37:39.922902 3661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-1/wal
uuid: "c21c5a2c8f6d4fb184eb93acd9821484"
format_stamp: "Formatted at 2026-06-19 14:37:39 on dist-test-slave-32hj"
server_key: "e881b393d57dbc95b9dd0aa27d645bd5"
server_key_iv: "cdcf3dc9d955d374e660eac43872de7b"
server_key_version: "encryptionkey@0"
I20260619 14:37:39.922992 3661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:37:39.936532 3661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:37:39.939224 3661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:37:39.939383 3661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:37:39.939898 3661 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:37:39.940663 3661 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:37:39.940707 3661 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:39.940794 3661 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:37:39.940820 3661 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:39.948784 3661 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38435
I20260619 14:37:39.948849 3789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38435 every 8 connection(s)
I20260619 14:37:39.949626 3661 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:37:39.954614 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 3661
I20260619 14:37:39.954722 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:37:39.954975 392 external_mini_cluster.cc:1428] Setting key c2ab99b9ff5796bf93f72088574e71ff
Jun 19 14:37:39 dist-test-slave-32hj krb5kdc[3420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879859, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:37:39.963382 3504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:39.951245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35725 (local address 127.0.98.62:40527)
0619 14:37:39.951369 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:39.951373 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:39.952172 (+ 799us) server_negotiation.cc:407] Connection header received
0619 14:37:39.953493 (+ 1321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:39.953496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:39.953551 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:39.953624 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:39.954721 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:39.955438 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:39.956922 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:39.957229 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:39.957328 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:39.959927 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:39.959931 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:39.959933 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:39.959955 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:39.961250 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:39.961821 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:39.961824 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:39.961825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:39.961890 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:39.962231 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:39.962234 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:39.962235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:39.962406 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:39.962478 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:39.963129 (+ 651us) server_negotiation.cc:299] Negotiation successful
0619 14:37:39.963239 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 14:37:39.964447 3792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:39.951605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40527 (local address 127.0.98.2:35725)
0619 14:37:39.952028 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:39.952070 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:39.953301 (+ 1231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:39.953780 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:39.953787 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:39.954126 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:39.954583 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:39.954596 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:39.955596 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:39.955600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:39.956815 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:39.956824 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:39.957351 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:39.957354 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:39.957491 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:39.958059 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:39.958079 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:39.959776 (+ 1697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:39.961361 (+ 1585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:39.961368 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:39.961383 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:39.961735 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:39.961963 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:39.961966 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:39.961968 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:39.962140 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:39.962501 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:39.962509 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:39.962820 (+ 311us) client_negotiation.cc:769] Sending connection context
0619 14:37:39.963078 (+ 258us) client_negotiation.cc:240] Negotiation successful
0619 14:37:39.963404 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":117,"threads_started":1}
I20260619 14:37:39.965564 3790 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40527
I20260619 14:37:39.965765 3790 heartbeater.cc:464] Registering TS with master...
I20260619 14:37:39.966223 3790 heartbeater.cc:511] Master 127.0.98.62:40527 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 14:37:39.967242 3465 ts_manager.cc:194] Registered new tserver with Master: c21c5a2c8f6d4fb184eb93acd9821484 (127.0.98.2:38435)
I20260619 14:37:39.967787 3465 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35725
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 14:37:39.980135 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40527
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:37:40.085368 3795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:37:40.085600 3795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:37:40.085640 3795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:37:40.089521 3795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:37:40.089571 3795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:37:40.089637 3795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:37:40.094774 3795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:37:40.095743 3795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:37:40.096740 3795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:37:40.105672 3795 server_base.cc:1034] running on GCE node
W20260619 14:37:40.105638 3801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:40.105628 3800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:40.105638 3803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:40.106225 3795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:37:40.106810 3795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:37:40.107956 3795 hybrid_clock.cc:648] HybridClock initialized: now 1781879860107926 us; error 42 us; skew 500 ppm
Jun 19 14:37:40 dist-test-slave-32hj krb5kdc[3420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879860, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:37:40.114013 3795 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:37:40.114930 3795 webserver.cc:460] Webserver started at http://127.0.98.3:39843/ using document root <none> and password file <none>
I20260619 14:37:40.115440 3795 fs_manager.cc:359] Metadata directory not provided
I20260619 14:37:40.115501 3795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:37:40.115661 3795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:37:40.117687 3795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-2/data/instance:
uuid: "17a9cb52efcd4638a746ef4f33710a1b"
format_stamp: "Formatted at 2026-06-19 14:37:40 on dist-test-slave-32hj"
server_key: "767b4267c0f90e9ff27749907bb9e338"
server_key_iv: "d886513c6680358b0bbfc321b3a928cf"
server_key_version: "encryptionkey@0"
I20260619 14:37:40.118198 3795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-2/wal/instance:
uuid: "17a9cb52efcd4638a746ef4f33710a1b"
format_stamp: "Formatted at 2026-06-19 14:37:40 on dist-test-slave-32hj"
server_key: "767b4267c0f90e9ff27749907bb9e338"
server_key_iv: "d886513c6680358b0bbfc321b3a928cf"
server_key_version: "encryptionkey@0"
I20260619 14:37:40.121841 3795 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:37:40.124243 3810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:40.125542 3795 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:37:40.125643 3795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-2/wal
uuid: "17a9cb52efcd4638a746ef4f33710a1b"
format_stamp: "Formatted at 2026-06-19 14:37:40 on dist-test-slave-32hj"
server_key: "767b4267c0f90e9ff27749907bb9e338"
server_key_iv: "d886513c6680358b0bbfc321b3a928cf"
server_key_version: "encryptionkey@0"
I20260619 14:37:40.125739 3795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:37:40.170315 3795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:37:40.173203 3795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:37:40.173372 3795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:37:40.173941 3795 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:37:40.174722 3795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:37:40.174767 3795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:40.174818 3795 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:37:40.174844 3795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:40.182724 3795 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34057
I20260619 14:37:40.182791 3923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34057 every 8 connection(s)
I20260619 14:37:40.183590 3795 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:37:40.185906 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 3795
I20260619 14:37:40.186017 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781879837648036-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:37:40.186264 392 external_mini_cluster.cc:1428] Setting key 5c51684dead324b5d85d63ba5193c912
Jun 19 14:37:40 dist-test-slave-32hj krb5kdc[3420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879860, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:37:40.196120 3504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:40.185399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36103 (local address 127.0.98.62:40527)
0619 14:37:40.185536 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:40.185540 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:40.186411 (+ 871us) server_negotiation.cc:407] Connection header received
0619 14:37:40.187882 (+ 1471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:40.187885 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:40.187927 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:40.187981 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:40.189158 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:40.189670 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:40.190705 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:40.190900 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:40.190965 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:40.193162 (+ 2197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:40.193166 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:40.193168 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:40.193192 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:40.194482 (+ 1290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:40.194909 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:40.194911 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:40.194912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:40.194987 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:40.195269 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:40.195271 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:40.195272 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:40.195433 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:40.195503 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:40.195895 (+ 392us) server_negotiation.cc:299] Negotiation successful
0619 14:37:40.195983 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 14:37:40.196893 3926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:40.185781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40527 (local address 127.0.98.3:36103)
0619 14:37:40.186241 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:40.186295 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:40.187675 (+ 1380us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:40.188147 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:40.188154 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:40.188542 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:40.189046 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:40.189059 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:40.189758 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:40.189760 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:40.190620 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:40.190628 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:40.190998 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:40.191000 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:40.191131 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:40.191673 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:40.191689 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:40.193024 (+ 1335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:40.194583 (+ 1559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:40.194587 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:40.194598 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:40.194828 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:40.195086 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:40.195088 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:40.195089 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:40.195205 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:40.195501 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:40.195505 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:40.195724 (+ 219us) client_negotiation.cc:769] Sending connection context
0619 14:37:40.195866 (+ 142us) client_negotiation.cc:240] Negotiation successful
0619 14:37:40.196098 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":114,"threads_started":1}
I20260619 14:37:40.198056 3924 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40527
I20260619 14:37:40.198272 3924 heartbeater.cc:464] Registering TS with master...
I20260619 14:37:40.198774 3924 heartbeater.cc:511] Master 127.0.98.62:40527 requested a full tablet report, sending...
I20260619 14:37:40.199733 3464 ts_manager.cc:194] Registered new tserver with Master: 17a9cb52efcd4638a746ef4f33710a1b (127.0.98.3:34057)
I20260619 14:37:40.200309 3464 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36103
I20260619 14:37:40.211253 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 14:37:40 dist-test-slave-32hj krb5kdc[3420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879860, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 19 14:37:40 dist-test-slave-32hj krb5kdc[3420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879860, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:37:40.233794 3504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:40.226309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33426 (local address 127.0.98.62:40527)
0619 14:37:40.226441 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:40.226445 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:40.226583 (+ 138us) server_negotiation.cc:407] Connection header received
0619 14:37:40.226709 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:40.226712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:40.226758 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:40.226828 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:40.227442 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:40.227952 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:40.228796 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:40.229043 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:40.229144 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:40.231306 (+ 2162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:40.231309 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:40.231311 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:40.231334 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:40.232430 (+ 1096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:40.232817 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:40.232820 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:40.232822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:40.232905 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:40.233123 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:40.233125 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:40.233126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:40.233318 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:40.233420 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:40.233551 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 14:37:40.233633 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260619 14:37:40.234122 3464 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:33426
I20260619 14:37:40.234890 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 3527
I20260619 14:37:40.239774 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 3661
I20260619 14:37:40.244241 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 3795
I20260619 14:37:40.248210 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 3433
2026-06-19T14:37:40Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1981 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:37:40 dist-test-slave-32hj krb5kdc[3936](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:37:40 dist-test-slave-32hj krb5kdc[3936](info): set up 2 sockets
Jun 19 14:37:40 dist-test-slave-32hj krb5kdc[3936](info): commencing operation
krb5kdc: starting...
W20260619 14:37:41.200404 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/krb5kdc/test-user.keytab.
Jun 19 14:37:41 dist-test-slave-32hj krb5kdc[3936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:37:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:37:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:37:41.307130 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36725
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36725
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:37:41.411675 3949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:37:41.411931 3949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:37:41.411985 3949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:37:41.415841 3949 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:37:41.415905 3949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:37:41.415931 3949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:37:41.415951 3949 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:37:41.415972 3949 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:37:41.420989 3949 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:37:41.421932 3949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:37:41.422873 3949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:37:41.430693 3955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:41.430711 3957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:41.430874 3949 server_base.cc:1034] running on GCE node
W20260619 14:37:41.430724 3954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:41.431488 3949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:37:41.432511 3949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:37:41.433694 3949 hybrid_clock.cc:648] HybridClock initialized: now 1781879861433661 us; error 45 us; skew 500 ppm
Jun 19 14:37:41 dist-test-slave-32hj krb5kdc[3936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879861, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:37:41.436200 3949 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:37:41.437101 3949 webserver.cc:460] Webserver started at http://127.0.98.62:45329/ using document root <none> and password file <none>
I20260619 14:37:41.437608 3949 fs_manager.cc:359] Metadata directory not provided
I20260619 14:37:41.437671 3949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:37:41.437850 3949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:37:41.439759 3949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/master-0/data/instance:
uuid: "4df9b697a7714662b08fa93f94cb744b"
format_stamp: "Formatted at 2026-06-19 14:37:41 on dist-test-slave-32hj"
server_key: "4defb19f0aecf0f0be97f6d4cfb3762e"
server_key_iv: "c15d4cfc7f56d45d41a3803d0b36ee2c"
server_key_version: "encryptionkey@0"
I20260619 14:37:41.440264 3949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/master-0/wal/instance:
uuid: "4df9b697a7714662b08fa93f94cb744b"
format_stamp: "Formatted at 2026-06-19 14:37:41 on dist-test-slave-32hj"
server_key: "4defb19f0aecf0f0be97f6d4cfb3762e"
server_key_iv: "c15d4cfc7f56d45d41a3803d0b36ee2c"
server_key_version: "encryptionkey@0"
I20260619 14:37:41.443717 3949 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 14:37:41.446107 3964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:41.447310 3949 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:37:41.447423 3949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/master-0/wal
uuid: "4df9b697a7714662b08fa93f94cb744b"
format_stamp: "Formatted at 2026-06-19 14:37:41 on dist-test-slave-32hj"
server_key: "4defb19f0aecf0f0be97f6d4cfb3762e"
server_key_iv: "c15d4cfc7f56d45d41a3803d0b36ee2c"
server_key_version: "encryptionkey@0"
I20260619 14:37:41.447535 3949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:37:41.464668 3949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:37:41.467417 3949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:37:41.467592 3949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:37:41.474164 3949 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36725
I20260619 14:37:41.474180 4016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36725 every 8 connection(s)
I20260619 14:37:41.475088 3949 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:37:41.477782 4017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:41.482203 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 3949
I20260619 14:37:41.482340 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/master-0/wal/instance
I20260619 14:37:41.482614 392 external_mini_cluster.cc:1428] Setting key 67c59bb520c6dada94bddcfee5995c04
I20260619 14:37:41.483264 4017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4df9b697a7714662b08fa93f94cb744b: Bootstrap starting.
I20260619 14:37:41.485561 4017 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4df9b697a7714662b08fa93f94cb744b: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:41.486243 4017 log.cc:824] T 00000000000000000000000000000000 P 4df9b697a7714662b08fa93f94cb744b: Log is configured to *not* fsync() on all Append() calls
I20260619 14:37:41.488180 4017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4df9b697a7714662b08fa93f94cb744b: No bootstrap required, opened a new log
Jun 19 14:37:41 dist-test-slave-32hj krb5kdc[3936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:37:41.492074 4017 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4df9b697a7714662b08fa93f94cb744b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4df9b697a7714662b08fa93f94cb744b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36725 } }
I20260619 14:37:41.492321 4017 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4df9b697a7714662b08fa93f94cb744b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:41.492398 4017 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4df9b697a7714662b08fa93f94cb744b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4df9b697a7714662b08fa93f94cb744b, State: Initialized, Role: FOLLOWER
I20260619 14:37:41.492873 4017 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4df9b697a7714662b08fa93f94cb744b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4df9b697a7714662b08fa93f94cb744b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36725 } }
I20260619 14:37:41.492995 4017 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4df9b697a7714662b08fa93f94cb744b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:37:41.493053 4017 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4df9b697a7714662b08fa93f94cb744b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:37:41.493142 4017 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4df9b697a7714662b08fa93f94cb744b [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:41.493930 4017 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4df9b697a7714662b08fa93f94cb744b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4df9b697a7714662b08fa93f94cb744b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36725 } }
I20260619 14:37:41.494263 4017 leader_election.cc:302] T 00000000000000000000000000000000 P 4df9b697a7714662b08fa93f94cb744b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4df9b697a7714662b08fa93f94cb744b; no voters:
I20260619 14:37:41.494526 4017 leader_election.cc:288] T 00000000000000000000000000000000 P 4df9b697a7714662b08fa93f94cb744b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:37:41.494864 4023 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4df9b697a7714662b08fa93f94cb744b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:37:41.495033 4023 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4df9b697a7714662b08fa93f94cb744b [term 1 LEADER]: Becoming Leader. State: Replica: 4df9b697a7714662b08fa93f94cb744b, State: Running, Role: LEADER
I20260619 14:37:41.495360 4023 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4df9b697a7714662b08fa93f94cb744b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4df9b697a7714662b08fa93f94cb744b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36725 } }
I20260619 14:37:41.495738 4017 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4df9b697a7714662b08fa93f94cb744b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:37:41.496515 4020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:41.483951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48366 (local address 127.0.98.62:36725)
0619 14:37:41.484373 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:41.484387 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:41.484423 (+ 36us) server_negotiation.cc:407] Connection header received
0619 14:37:41.485076 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:41.485093 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:41.485417 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:41.485714 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:41.486496 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:41.487788 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:41.488654 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:41.489330 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:41.489480 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:41.491901 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:41.491907 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:41.491922 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:41.491951 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:41.493718 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:41.494141 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:41.494147 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:41.494153 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:41.494262 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:41.494540 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:41.494544 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:41.494545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:41.494966 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:41.495139 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:41.495416 (+ 277us) server_negotiation.cc:299] Negotiation successful
0619 14:37:41.495673 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":129,"threads_started":1}
I20260619 14:37:41.496572 4023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4df9b697a7714662b08fa93f94cb744b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4df9b697a7714662b08fa93f94cb744b. Latest consensus state: current_term: 1 leader_uuid: "4df9b697a7714662b08fa93f94cb744b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4df9b697a7714662b08fa93f94cb744b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36725 } } }
I20260619 14:37:41.496676 4023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4df9b697a7714662b08fa93f94cb744b [sys.catalog]: This master's current role is: LEADER
I20260619 14:37:41.497037 4031 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:37:41.496989 4022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4df9b697a7714662b08fa93f94cb744b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4df9b697a7714662b08fa93f94cb744b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4df9b697a7714662b08fa93f94cb744b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36725 } } }
I20260619 14:37:41.497118 4022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4df9b697a7714662b08fa93f94cb744b [sys.catalog]: This master's current role is: LEADER
I20260619 14:37:41.500068 4031 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:37:41.504951 4031 catalog_manager.cc:1347] Generated new cluster ID: 68473967c0444d4ca0c1733135c9fb05
I20260619 14:37:41.505010 4031 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:37:41.533025 4031 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:37:41.534283 4031 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:37:41.544849 4031 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4df9b697a7714662b08fa93f94cb744b: Generated new TSK 0
I20260619 14:37:41.545643 4031 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 14:37:41.569638 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36725
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:37:41.674876 4043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:37:41.675122 4043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:37:41.675163 4043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:37:41.679118 4043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:37:41.679170 4043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:37:41.679225 4043 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:37:41.684219 4043 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4043
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:37:41.685230 4043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:37:41.686174 4043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:37:41.695034 4048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:41.695079 4043 server_base.cc:1034] running on GCE node
W20260619 14:37:41.695115 4049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:41.695258 4051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:41.695695 4043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:37:41.696377 4043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:37:41.697574 4043 hybrid_clock.cc:648] HybridClock initialized: now 1781879861697555 us; error 37 us; skew 500 ppm
Jun 19 14:37:41 dist-test-slave-32hj krb5kdc[3936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879861, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:37:41.700130 4043 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:37:41.700941 4043 webserver.cc:460] Webserver started at http://127.0.98.1:40245/ using document root <none> and password file <none>
I20260619 14:37:41.701431 4043 fs_manager.cc:359] Metadata directory not provided
I20260619 14:37:41.701491 4043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:37:41.701685 4043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:37:41.703637 4043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/ts-0/data/instance:
uuid: "8197e8b451b445f3ab17753442a2a1e2"
format_stamp: "Formatted at 2026-06-19 14:37:41 on dist-test-slave-32hj"
server_key: "cef8f0952451323e4e52257cfa40518c"
server_key_iv: "b79fd68314b8e5e3d28771c1e79acd56"
server_key_version: "encryptionkey@0"
I20260619 14:37:41.704149 4043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8197e8b451b445f3ab17753442a2a1e2"
format_stamp: "Formatted at 2026-06-19 14:37:41 on dist-test-slave-32hj"
server_key: "cef8f0952451323e4e52257cfa40518c"
server_key_iv: "b79fd68314b8e5e3d28771c1e79acd56"
server_key_version: "encryptionkey@0"
I20260619 14:37:41.707568 4043 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 14:37:41.709772 4058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:41.710855 4043 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:37:41.710961 4043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/ts-0/wal
uuid: "8197e8b451b445f3ab17753442a2a1e2"
format_stamp: "Formatted at 2026-06-19 14:37:41 on dist-test-slave-32hj"
server_key: "cef8f0952451323e4e52257cfa40518c"
server_key_iv: "b79fd68314b8e5e3d28771c1e79acd56"
server_key_version: "encryptionkey@0"
I20260619 14:37:41.711043 4043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:37:41.734225 4043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:37:41.737313 4043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:37:41.737488 4043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:37:41.738067 4043 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:37:41.738868 4043 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:37:41.738914 4043 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:41.738967 4043 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:37:41.738994 4043 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:41.747330 4043 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46757
I20260619 14:37:41.747370 4171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46757 every 8 connection(s)
I20260619 14:37:41.748168 4043 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:37:41.755364 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 4043
I20260619 14:37:41.755465 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:37:41.755753 392 external_mini_cluster.cc:1428] Setting key e4d2dabf0e7b181464780f56d06a7ba6
Jun 19 14:37:41 dist-test-slave-32hj krb5kdc[3936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879861, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:37:41.761842 4020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:41.750083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49143 (local address 127.0.98.62:36725)
0619 14:37:41.750195 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:41.750199 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:41.750956 (+ 757us) server_negotiation.cc:407] Connection header received
0619 14:37:41.752351 (+ 1395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:41.752354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:41.752411 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:41.752501 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:41.753661 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:41.754176 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:41.755303 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:41.755591 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:41.755678 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:41.758523 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:41.758526 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:41.758528 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:41.758550 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:41.759946 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:41.760430 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:41.760432 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:41.760434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:41.760515 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:41.760847 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:41.760850 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:41.760851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:41.761021 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:41.761106 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:41.761570 (+ 464us) server_negotiation.cc:299] Negotiation successful
0619 14:37:41.761689 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 14:37:41.762615 4174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:41.750402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36725 (local address 127.0.98.1:49143)
0619 14:37:41.750808 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:41.750850 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:41.752157 (+ 1307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:41.752688 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:41.752694 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:41.753072 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:41.753527 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:41.753539 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:41.754286 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:41.754289 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:41.755163 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:41.755175 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:41.755754 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:41.755758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:41.755962 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:41.756588 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:41.756610 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:41.758367 (+ 1757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:41.760059 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:41.760063 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:41.760074 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:41.760318 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:41.760599 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:41.760602 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:41.760604 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:41.760755 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:41.761116 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:41.761123 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:41.761343 (+ 220us) client_negotiation.cc:769] Sending connection context
0619 14:37:41.761501 (+ 158us) client_negotiation.cc:240] Negotiation successful
0619 14:37:41.761809 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":111,"threads_started":1}
I20260619 14:37:41.763696 4172 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36725
I20260619 14:37:41.763904 4172 heartbeater.cc:464] Registering TS with master...
I20260619 14:37:41.764333 4172 heartbeater.cc:511] Master 127.0.98.62:36725 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 14:37:41.765736 3981 ts_manager.cc:194] Registered new tserver with Master: 8197e8b451b445f3ab17753442a2a1e2 (127.0.98.1:46757)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 14:37:41.766929 3981 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49143
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 14:37:41.780349 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36725
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:37:41.882579 4177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:37:41.882795 4177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:37:41.882833 4177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:37:41.886689 4177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:37:41.886736 4177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:37:41.886790 4177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:37:41.891656 4177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:37:41.892510 4177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:37:41.893404 4177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:37:41.902129 4182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:41.902143 4183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:41.902194 4185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:41.902248 4177 server_base.cc:1034] running on GCE node
I20260619 14:37:41.902726 4177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:37:41.903254 4177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:37:41.904405 4177 hybrid_clock.cc:648] HybridClock initialized: now 1781879861904387 us; error 29 us; skew 500 ppm
Jun 19 14:37:41 dist-test-slave-32hj krb5kdc[3936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879861, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:37:41.906873 4177 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:37:41.907661 4177 webserver.cc:460] Webserver started at http://127.0.98.2:45379/ using document root <none> and password file <none>
I20260619 14:37:41.908152 4177 fs_manager.cc:359] Metadata directory not provided
I20260619 14:37:41.908213 4177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:37:41.908420 4177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:37:41.910386 4177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/ts-1/data/instance:
uuid: "e6116de56a53413d8068f1115d7a29ec"
format_stamp: "Formatted at 2026-06-19 14:37:41 on dist-test-slave-32hj"
server_key: "3dce3db5c1665b205586ef4bdbb9079f"
server_key_iv: "6e64a1699b49bfe2deae6b6e2e6a64b6"
server_key_version: "encryptionkey@0"
I20260619 14:37:41.910888 4177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e6116de56a53413d8068f1115d7a29ec"
format_stamp: "Formatted at 2026-06-19 14:37:41 on dist-test-slave-32hj"
server_key: "3dce3db5c1665b205586ef4bdbb9079f"
server_key_iv: "6e64a1699b49bfe2deae6b6e2e6a64b6"
server_key_version: "encryptionkey@0"
I20260619 14:37:41.914312 4177 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:37:41.916553 4192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:41.917758 4177 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:37:41.917872 4177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/ts-1/wal
uuid: "e6116de56a53413d8068f1115d7a29ec"
format_stamp: "Formatted at 2026-06-19 14:37:41 on dist-test-slave-32hj"
server_key: "3dce3db5c1665b205586ef4bdbb9079f"
server_key_iv: "6e64a1699b49bfe2deae6b6e2e6a64b6"
server_key_version: "encryptionkey@0"
I20260619 14:37:41.917955 4177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:37:41.929237 4177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:37:41.931949 4177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:37:41.932101 4177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:37:41.932613 4177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:37:41.933368 4177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:37:41.933410 4177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:41.933449 4177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:37:41.933478 4177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:41.941169 4177 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37635
I20260619 14:37:41.941218 4305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37635 every 8 connection(s)
I20260619 14:37:41.941993 4177 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:37:41.945510 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 4177
I20260619 14:37:41.945637 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:37:41.945880 392 external_mini_cluster.cc:1428] Setting key 17e4179feb4c710a7facc561f1932db5
Jun 19 14:37:41 dist-test-slave-32hj krb5kdc[3936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879861, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 14:37:41.955459 4020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:41.943629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48795 (local address 127.0.98.62:36725)
0619 14:37:41.943746 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:41.943750 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:41.944585 (+ 835us) server_negotiation.cc:407] Connection header received
0619 14:37:41.946033 (+ 1448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:41.946036 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:41.946094 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:41.946164 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:41.947266 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:41.947761 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:41.949184 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:41.949395 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:41.949480 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:41.952127 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:41.952130 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:41.952132 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:41.952155 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:41.953464 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:41.954009 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:41.954012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:41.954013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:41.954086 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:41.954427 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:41.954430 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:41.954431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:41.954609 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:41.954683 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:41.955169 (+ 486us) server_negotiation.cc:299] Negotiation successful
0619 14:37:41.955276 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 14:37:41.956112 4308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:41.943980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36725 (local address 127.0.98.2:48795)
0619 14:37:41.944429 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:41.944471 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:41.945805 (+ 1334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:41.946333 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:41.946348 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:41.946677 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:41.947135 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:41.947148 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:41.947870 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:41.947873 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:41.949072 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:41.949083 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:41.949536 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:41.949540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:41.949716 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:41.950284 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:41.950303 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:41.951957 (+ 1654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:41.953569 (+ 1612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:41.953575 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:41.953589 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:41.953918 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:41.954169 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:41.954173 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:41.954175 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:41.954339 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:41.954701 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:41.954708 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:41.954928 (+ 220us) client_negotiation.cc:769] Sending connection context
0619 14:37:41.955077 (+ 149us) client_negotiation.cc:240] Negotiation successful
0619 14:37:41.955308 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":109,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 14:37:41.957379 4306 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36725
I20260619 14:37:41.957581 4306 heartbeater.cc:464] Registering TS with master...
I20260619 14:37:41.958034 4306 heartbeater.cc:511] Master 127.0.98.62:36725 requested a full tablet report, sending...
I20260619 14:37:41.959034 3981 ts_manager.cc:194] Registered new tserver with Master: e6116de56a53413d8068f1115d7a29ec (127.0.98.2:37635)
I20260619 14:37:41.959718 3981 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48795
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 14:37:41.971012 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36725
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:37:42.077000 4311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:37:42.077265 4311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:37:42.077306 4311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:37:42.081324 4311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:37:42.081387 4311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:37:42.081445 4311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:37:42.086566 4311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:37:42.087574 4311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:37:42.088550 4311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:37:42.097587 4319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:42.097661 4317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:42.097729 4311 server_base.cc:1034] running on GCE node
W20260619 14:37:42.097661 4316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:42.098187 4311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:37:42.098862 4311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:37:42.100018 4311 hybrid_clock.cc:648] HybridClock initialized: now 1781879862099993 us; error 37 us; skew 500 ppm
Jun 19 14:37:42 dist-test-slave-32hj krb5kdc[3936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879862, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:37:42.102516 4311 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:37:42.103243 4311 webserver.cc:460] Webserver started at http://127.0.98.3:34411/ using document root <none> and password file <none>
I20260619 14:37:42.103732 4311 fs_manager.cc:359] Metadata directory not provided
I20260619 14:37:42.103788 4311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:37:42.103969 4311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:37:42.105870 4311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/ts-2/data/instance:
uuid: "9b0cb3252caf43d49448e0b592095b79"
format_stamp: "Formatted at 2026-06-19 14:37:42 on dist-test-slave-32hj"
server_key: "ca8da2fd303ae87aaf4840dbd208ac61"
server_key_iv: "31c61065a8458ddeeacd518c5ce11cd4"
server_key_version: "encryptionkey@0"
I20260619 14:37:42.106384 4311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9b0cb3252caf43d49448e0b592095b79"
format_stamp: "Formatted at 2026-06-19 14:37:42 on dist-test-slave-32hj"
server_key: "ca8da2fd303ae87aaf4840dbd208ac61"
server_key_iv: "31c61065a8458ddeeacd518c5ce11cd4"
server_key_version: "encryptionkey@0"
I20260619 14:37:42.109707 4311 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:37:42.111968 4326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:42.113181 4311 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:37:42.113282 4311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/ts-2/wal
uuid: "9b0cb3252caf43d49448e0b592095b79"
format_stamp: "Formatted at 2026-06-19 14:37:42 on dist-test-slave-32hj"
server_key: "ca8da2fd303ae87aaf4840dbd208ac61"
server_key_iv: "31c61065a8458ddeeacd518c5ce11cd4"
server_key_version: "encryptionkey@0"
I20260619 14:37:42.113379 4311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:37:42.132169 4311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:37:42.134824 4311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:37:42.134992 4311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:37:42.135546 4311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:37:42.136325 4311 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:37:42.136370 4311 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:42.136420 4311 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:37:42.136445 4311 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:42.144640 4311 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35375
I20260619 14:37:42.144675 4439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35375 every 8 connection(s)
I20260619 14:37:42.145511 4311 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:37:42.146396 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 4311
I20260619 14:37:42.146507 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781879837648036-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:37:42.146773 392 external_mini_cluster.cc:1428] Setting key e0a788d71a10c25085626af1f822864b
Jun 19 14:37:42 dist-test-slave-32hj krb5kdc[3936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879862, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:37:42.158352 4020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:42.147306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59931 (local address 127.0.98.62:36725)
0619 14:37:42.147440 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:42.147445 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:42.148473 (+ 1028us) server_negotiation.cc:407] Connection header received
0619 14:37:42.149864 (+ 1391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:42.149867 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:42.149919 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:42.150020 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:42.151169 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:42.151747 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:42.152876 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:42.153082 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:42.153167 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:42.155344 (+ 2177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:42.155347 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:42.155349 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:42.155380 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:42.156439 (+ 1059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:42.156945 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:42.156948 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:42.156949 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:42.157021 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:42.157331 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:42.157334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:42.157335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:42.157508 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:42.157595 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:42.158079 (+ 484us) server_negotiation.cc:299] Negotiation successful
0619 14:37:42.158193 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 14:37:42.159087 4442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:42.147747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36725 (local address 127.0.98.3:59931)
0619 14:37:42.148336 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:42.148381 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:42.149676 (+ 1295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:42.150174 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:42.150181 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:42.150565 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:42.151041 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:42.151054 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:42.151860 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:42.151868 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:42.152766 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:42.152775 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:42.153207 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:42.153211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:42.153374 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:42.153884 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:42.153901 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:42.155200 (+ 1299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:42.156536 (+ 1336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:42.156542 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:42.156555 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:42.156848 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:42.157099 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:42.157102 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:42.157104 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:42.157237 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:42.157604 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:42.157611 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:42.157847 (+ 236us) client_negotiation.cc:769] Sending connection context
0619 14:37:42.158007 (+ 160us) client_negotiation.cc:240] Negotiation successful
0619 14:37:42.158305 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":158,"threads_started":1}
I20260619 14:37:42.160135 4440 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36725
I20260619 14:37:42.160388 4440 heartbeater.cc:464] Registering TS with master...
I20260619 14:37:42.160889 4440 heartbeater.cc:511] Master 127.0.98.62:36725 requested a full tablet report, sending...
I20260619 14:37:42.161793 3980 ts_manager.cc:194] Registered new tserver with Master: 9b0cb3252caf43d49448e0b592095b79 (127.0.98.3:35375)
I20260619 14:37:42.162338 3980 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59931
I20260619 14:37:42.172091 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 14:37:42 dist-test-slave-32hj krb5kdc[3936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 14:37:42 dist-test-slave-32hj krb5kdc[3936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 14:37:42.196120 4447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:42.186804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56964 (local address 127.0.98.1:46757)
0619 14:37:42.187134 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:42.187145 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:42.187166 (+ 21us) server_negotiation.cc:407] Connection header received
0619 14:37:42.187211 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:42.187215 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:42.187395 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:42.187523 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:42.188179 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:42.188977 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:42.189802 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:42.190170 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:42.190262 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:42.192591 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:42.192599 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:42.192612 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:42.192636 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:42.194223 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:42.194785 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:42.194791 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:42.194792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:42.194878 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:42.195163 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:42.195166 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:42.195167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:42.195439 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:42.195618 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:42.195762 (+ 144us) server_negotiation.cc:299] Negotiation successful
0619 14:37:42.195901 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":99,"threads_started":1}
Jun 19 14:37:42 dist-test-slave-32hj krb5kdc[3936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:37:42.205889 4020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:42.198881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48368 (local address 127.0.98.62:36725)
0619 14:37:42.199024 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:42.199028 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:42.199042 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:37:42.199164 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:42.199168 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:42.199207 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:42.199267 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:42.199821 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:42.200321 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:42.201103 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:42.201311 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:42.201397 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:42.203413 (+ 2016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:42.203415 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:42.203417 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:42.203441 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:42.204608 (+ 1167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:42.204980 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:42.204982 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:42.204984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:42.205053 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:42.205235 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:42.205237 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:42.205238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:42.205437 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:42.205532 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:42.205682 (+ 150us) server_negotiation.cc:299] Negotiation successful
0619 14:37:42.205766 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260619 14:37:42.206245 3980 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48368
I20260619 14:37:42.206789 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 4043
I20260619 14:37:42.211714 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 4177
I20260619 14:37:42.216068 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 4311
I20260619 14:37:42.220167 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 3949
2026-06-19T14:37:42Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1972 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:37:42 dist-test-slave-32hj krb5kdc[4452](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:37:42 dist-test-slave-32hj krb5kdc[4452](info): set up 2 sockets
Jun 19 14:37:42 dist-test-slave-32hj krb5kdc[4452](info): commencing operation
krb5kdc: starting...
W20260619 14:37:43.164567 392 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/krb5kdc/test-user.keytab.
Jun 19 14:37:43 dist-test-slave-32hj krb5kdc[4452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:37:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:37:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:37:43.271468 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37313
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37313
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:37:43.377712 4465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:37:43.377947 4465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:37:43.378000 4465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:37:43.381903 4465 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:37:43.381955 4465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:37:43.381973 4465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:37:43.382004 4465 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:37:43.382030 4465 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:37:43.387171 4465 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:37:43.388156 4465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:37:43.389147 4465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:37:43.396610 4471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:43.396654 4470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:43.396759 4465 server_base.cc:1034] running on GCE node
W20260619 14:37:43.396631 4473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:43.397379 4465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:37:43.398401 4465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:37:43.399546 4465 hybrid_clock.cc:648] HybridClock initialized: now 1781879863399532 us; error 29 us; skew 500 ppm
Jun 19 14:37:43 dist-test-slave-32hj krb5kdc[4452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879863, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:37:43.402064 4465 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:37:43.402717 4465 fs_manager.cc:359] Metadata directory not provided
I20260619 14:37:43.402776 4465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:37:43.402964 4465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:37:43.404901 4465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/master-0/data/instance:
uuid: "a227e022210d46c6b8f11837b3b09f0e"
format_stamp: "Formatted at 2026-06-19 14:37:43 on dist-test-slave-32hj"
server_key: "806ec5380a7c412c1628349fbd93ac81"
server_key_iv: "78d7f47c7f6bc1a04cb7efcf5abd18c8"
server_key_version: "encryptionkey@0"
I20260619 14:37:43.405417 4465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/master-0/wal/instance:
uuid: "a227e022210d46c6b8f11837b3b09f0e"
format_stamp: "Formatted at 2026-06-19 14:37:43 on dist-test-slave-32hj"
server_key: "806ec5380a7c412c1628349fbd93ac81"
server_key_iv: "78d7f47c7f6bc1a04cb7efcf5abd18c8"
server_key_version: "encryptionkey@0"
I20260619 14:37:43.408869 4465 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:37:43.410951 4479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:43.412101 4465 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:37:43.412210 4465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/master-0/wal
uuid: "a227e022210d46c6b8f11837b3b09f0e"
format_stamp: "Formatted at 2026-06-19 14:37:43 on dist-test-slave-32hj"
server_key: "806ec5380a7c412c1628349fbd93ac81"
server_key_iv: "78d7f47c7f6bc1a04cb7efcf5abd18c8"
server_key_version: "encryptionkey@0"
I20260619 14:37:43.412297 4465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:37:43.443547 4465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:37:43.446488 4465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:37:43.446655 4465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:37:43.452797 4465 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37313
I20260619 14:37:43.453004 4531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37313 every 8 connection(s)
I20260619 14:37:43.453387 4465 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:37:43.455852 4532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:43.456895 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 4465
I20260619 14:37:43.457020 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/master-0/wal/instance
I20260619 14:37:43.457309 392 external_mini_cluster.cc:1428] Setting key aa44ef1220566b063c021eb597b986ab
I20260619 14:37:43.462024 4532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a227e022210d46c6b8f11837b3b09f0e: Bootstrap starting.
I20260619 14:37:43.464454 4532 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a227e022210d46c6b8f11837b3b09f0e: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:43.465173 4532 log.cc:824] T 00000000000000000000000000000000 P a227e022210d46c6b8f11837b3b09f0e: Log is configured to *not* fsync() on all Append() calls
Jun 19 14:37:43 dist-test-slave-32hj krb5kdc[4452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:37:43.467114 4532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a227e022210d46c6b8f11837b3b09f0e: No bootstrap required, opened a new log
I20260619 14:37:43.471149 4535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:43.458647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39896 (local address 127.0.98.62:37313)
0619 14:37:43.459028 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:43.459046 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:43.459088 (+ 42us) server_negotiation.cc:407] Connection header received
0619 14:37:43.459866 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:43.459889 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:43.460300 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:43.460647 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:43.461291 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:43.462664 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:43.463457 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:43.464110 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:43.464256 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:43.466553 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:43.466559 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:43.466574 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:43.466601 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:43.468319 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:43.468688 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:43.468694 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:43.468699 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:43.468847 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:43.469069 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:43.469072 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:43.469074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:43.469590 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:43.469766 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:43.469932 (+ 166us) server_negotiation.cc:299] Negotiation successful
0619 14:37:43.470216 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":97,"threads_started":1}
I20260619 14:37:43.471369 4532 raft_consensus.cc:348] T 00000000000000000000000000000000 P a227e022210d46c6b8f11837b3b09f0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a227e022210d46c6b8f11837b3b09f0e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37313 } }
I20260619 14:37:43.471596 4532 raft_consensus.cc:374] T 00000000000000000000000000000000 P a227e022210d46c6b8f11837b3b09f0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:43.471643 4532 raft_consensus.cc:729] T 00000000000000000000000000000000 P a227e022210d46c6b8f11837b3b09f0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a227e022210d46c6b8f11837b3b09f0e, State: Initialized, Role: FOLLOWER
I20260619 14:37:43.472040 4532 consensus_queue.cc:260] T 00000000000000000000000000000000 P a227e022210d46c6b8f11837b3b09f0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a227e022210d46c6b8f11837b3b09f0e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37313 } }
I20260619 14:37:43.472172 4532 raft_consensus.cc:388] T 00000000000000000000000000000000 P a227e022210d46c6b8f11837b3b09f0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:37:43.472235 4532 raft_consensus.cc:482] T 00000000000000000000000000000000 P a227e022210d46c6b8f11837b3b09f0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:37:43.472321 4532 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a227e022210d46c6b8f11837b3b09f0e [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:43.473343 4532 raft_consensus.cc:504] T 00000000000000000000000000000000 P a227e022210d46c6b8f11837b3b09f0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a227e022210d46c6b8f11837b3b09f0e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37313 } }
I20260619 14:37:43.473649 4532 leader_election.cc:302] T 00000000000000000000000000000000 P a227e022210d46c6b8f11837b3b09f0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a227e022210d46c6b8f11837b3b09f0e; no voters:
I20260619 14:37:43.473899 4532 leader_election.cc:288] T 00000000000000000000000000000000 P a227e022210d46c6b8f11837b3b09f0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:37:43.473990 4537 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a227e022210d46c6b8f11837b3b09f0e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:37:43.474174 4537 raft_consensus.cc:686] T 00000000000000000000000000000000 P a227e022210d46c6b8f11837b3b09f0e [term 1 LEADER]: Becoming Leader. State: Replica: a227e022210d46c6b8f11837b3b09f0e, State: Running, Role: LEADER
I20260619 14:37:43.474428 4537 consensus_queue.cc:237] T 00000000000000000000000000000000 P a227e022210d46c6b8f11837b3b09f0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a227e022210d46c6b8f11837b3b09f0e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37313 } }
I20260619 14:37:43.474936 4532 sys_catalog.cc:564] T 00000000000000000000000000000000 P a227e022210d46c6b8f11837b3b09f0e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:37:43.475404 4537 sys_catalog.cc:455] T 00000000000000000000000000000000 P a227e022210d46c6b8f11837b3b09f0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a227e022210d46c6b8f11837b3b09f0e. Latest consensus state: current_term: 1 leader_uuid: "a227e022210d46c6b8f11837b3b09f0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a227e022210d46c6b8f11837b3b09f0e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37313 } } }
I20260619 14:37:43.475587 4537 sys_catalog.cc:458] T 00000000000000000000000000000000 P a227e022210d46c6b8f11837b3b09f0e [sys.catalog]: This master's current role is: LEADER
I20260619 14:37:43.475771 4539 sys_catalog.cc:455] T 00000000000000000000000000000000 P a227e022210d46c6b8f11837b3b09f0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a227e022210d46c6b8f11837b3b09f0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a227e022210d46c6b8f11837b3b09f0e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37313 } } }
I20260619 14:37:43.475899 4539 sys_catalog.cc:458] T 00000000000000000000000000000000 P a227e022210d46c6b8f11837b3b09f0e [sys.catalog]: This master's current role is: LEADER
I20260619 14:37:43.476158 4544 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:37:43.478996 4544 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:37:43.484413 4544 catalog_manager.cc:1347] Generated new cluster ID: c086f799387b4c9eb43455d3ada0a3f3
I20260619 14:37:43.484467 4544 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:37:43.491819 4544 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:37:43.492745 4544 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:37:43.507148 4544 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a227e022210d46c6b8f11837b3b09f0e: Generated new TSK 0
I20260619 14:37:43.507767 4544 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 14:37:43.538705 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37313
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:37:43.644908 4558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:37:43.645179 4558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:37:43.645243 4558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:37:43.649246 4558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:37:43.649307 4558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:37:43.649389 4558 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:37:43.654578 4558 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:37:43.655577 4558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:37:43.656564 4558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:37:43.665485 4563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:43.665457 4564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:43.665655 4558 server_base.cc:1034] running on GCE node
W20260619 14:37:43.665473 4566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:43.666146 4558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:37:43.666721 4558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:37:43.667867 4558 hybrid_clock.cc:648] HybridClock initialized: now 1781879863667849 us; error 29 us; skew 500 ppm
Jun 19 14:37:43 dist-test-slave-32hj krb5kdc[4452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879863, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:37:43.670486 4558 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:37:43.671249 4558 fs_manager.cc:359] Metadata directory not provided
I20260619 14:37:43.671310 4558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:37:43.671494 4558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:37:43.673509 4558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/ts-0/data/instance:
uuid: "f4605c31c89d4b219a87d36179c22387"
format_stamp: "Formatted at 2026-06-19 14:37:43 on dist-test-slave-32hj"
server_key: "1dcca9f5d3e3adfa5c585040ae19b48d"
server_key_iv: "c2fc6b6c7f5042efb2869a837be36303"
server_key_version: "encryptionkey@0"
I20260619 14:37:43.674038 4558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f4605c31c89d4b219a87d36179c22387"
format_stamp: "Formatted at 2026-06-19 14:37:43 on dist-test-slave-32hj"
server_key: "1dcca9f5d3e3adfa5c585040ae19b48d"
server_key_iv: "c2fc6b6c7f5042efb2869a837be36303"
server_key_version: "encryptionkey@0"
I20260619 14:37:43.677732 4558 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:37:43.680006 4572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:43.681218 4558 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:37:43.681313 4558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/ts-0/wal
uuid: "f4605c31c89d4b219a87d36179c22387"
format_stamp: "Formatted at 2026-06-19 14:37:43 on dist-test-slave-32hj"
server_key: "1dcca9f5d3e3adfa5c585040ae19b48d"
server_key_iv: "c2fc6b6c7f5042efb2869a837be36303"
server_key_version: "encryptionkey@0"
I20260619 14:37:43.681406 4558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:37:43.710968 4558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:37:43.713783 4558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:37:43.713945 4558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:37:43.714489 4558 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:37:43.715350 4558 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:37:43.715394 4558 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:43.715435 4558 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:37:43.715471 4558 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:43.723368 4558 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36831
I20260619 14:37:43.723393 4685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36831 every 8 connection(s)
I20260619 14:37:43.723932 4558 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:37:43.724617 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 4558
I20260619 14:37:43.724746 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:37:43.725018 392 external_mini_cluster.cc:1428] Setting key 37e683dff9c987d076727a6a84339ea7
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Jun 19 14:37:43 dist-test-slave-32hj krb5kdc[4452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879863, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:37:43.739610 4535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:43.726309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42583 (local address 127.0.98.62:37313)
0619 14:37:43.726440 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:43.726444 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:43.727381 (+ 937us) server_negotiation.cc:407] Connection header received
0619 14:37:43.728843 (+ 1462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:43.728846 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:43.728900 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:43.728971 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:43.730098 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:43.730625 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:43.731704 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:43.731929 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:43.732016 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:43.736304 (+ 4288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:43.736307 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:43.736309 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:43.736331 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:43.737594 (+ 1263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:43.738135 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:43.738138 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:43.738139 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:43.738215 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:43.738545 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:43.738547 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:43.738548 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:43.738711 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:43.738792 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:43.739364 (+ 572us) server_negotiation.cc:299] Negotiation successful
0619 14:37:43.739470 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 14:37:43.740742 4688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:43.726787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37313 (local address 127.0.98.1:42583)
0619 14:37:43.727252 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:43.727292 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:43.728623 (+ 1331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:43.729160 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:43.729167 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:43.729504 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:43.729966 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:43.729980 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:43.730732 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:43.730736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:43.731593 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:43.731601 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:43.732059 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:43.732064 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:43.732216 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:43.732770 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:43.732788 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:43.736148 (+ 3360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:43.737700 (+ 1552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:43.737706 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:43.737721 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:43.738027 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:43.738292 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:43.738295 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:43.738297 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:43.738455 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:43.738806 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:43.738812 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:43.739130 (+ 318us) client_negotiation.cc:769] Sending connection context
0619 14:37:43.739337 (+ 207us) client_negotiation.cc:240] Negotiation successful
0619 14:37:43.739684 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":134,"threads_started":1}
I20260619 14:37:43.742053 4686 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37313
I20260619 14:37:43.742275 4686 heartbeater.cc:464] Registering TS with master...
I20260619 14:37:43.742751 4686 heartbeater.cc:511] Master 127.0.98.62:37313 requested a full tablet report, sending...
I20260619 14:37:43.744242 4496 ts_manager.cc:194] Registered new tserver with Master: f4605c31c89d4b219a87d36179c22387 (127.0.98.1:36831)
I20260619 14:37:43.745483 4496 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42583
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 14:37:43.749346 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37313
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:37:43.855931 4691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:37:43.856169 4691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:37:43.856236 4691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:37:43.860095 4691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:37:43.860154 4691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:37:43.860244 4691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:37:43.865176 4691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:37:43.866080 4691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:37:43.866989 4691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:37:43.875516 4699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:43.875540 4697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:43.875540 4696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:43.875725 4691 server_base.cc:1034] running on GCE node
I20260619 14:37:43.876157 4691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:37:43.876798 4691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:37:43.877949 4691 hybrid_clock.cc:648] HybridClock initialized: now 1781879863877932 us; error 29 us; skew 500 ppm
Jun 19 14:37:43 dist-test-slave-32hj krb5kdc[4452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879863, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:37:43.880437 4691 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:37:43.881116 4691 fs_manager.cc:359] Metadata directory not provided
I20260619 14:37:43.881176 4691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:37:43.881327 4691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:37:43.883164 4691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/ts-1/data/instance:
uuid: "7a1e11915e054aa887d3a603b64a486d"
format_stamp: "Formatted at 2026-06-19 14:37:43 on dist-test-slave-32hj"
server_key: "8c7fdecf742b1d5a50887ab1b92dfc22"
server_key_iv: "f483a8d8784e918aba9e4836abce2924"
server_key_version: "encryptionkey@0"
I20260619 14:37:43.883636 4691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7a1e11915e054aa887d3a603b64a486d"
format_stamp: "Formatted at 2026-06-19 14:37:43 on dist-test-slave-32hj"
server_key: "8c7fdecf742b1d5a50887ab1b92dfc22"
server_key_iv: "f483a8d8784e918aba9e4836abce2924"
server_key_version: "encryptionkey@0"
I20260619 14:37:43.887125 4691 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:37:43.889400 4705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:43.890600 4691 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 14:37:43.890693 4691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/ts-1/wal
uuid: "7a1e11915e054aa887d3a603b64a486d"
format_stamp: "Formatted at 2026-06-19 14:37:43 on dist-test-slave-32hj"
server_key: "8c7fdecf742b1d5a50887ab1b92dfc22"
server_key_iv: "f483a8d8784e918aba9e4836abce2924"
server_key_version: "encryptionkey@0"
I20260619 14:37:43.890794 4691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:37:43.907374 4691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:37:43.910360 4691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:37:43.910522 4691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:37:43.911063 4691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:37:43.911831 4691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:37:43.911882 4691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:43.911919 4691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:37:43.911933 4691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:43.920289 4691 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33365
I20260619 14:37:43.920311 4818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33365 every 8 connection(s)
I20260619 14:37:43.920876 4691 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:37:43.924419 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 4691
I20260619 14:37:43.924535 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:37:43.924822 392 external_mini_cluster.cc:1428] Setting key a655f4e55e0137707aa2509b9307d608
Jun 19 14:37:43 dist-test-slave-32hj krb5kdc[4452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879863, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 14:37:43.934370 4535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:43.922462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59405 (local address 127.0.98.62:37313)
0619 14:37:43.922580 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:43.922584 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:43.923422 (+ 838us) server_negotiation.cc:407] Connection header received
0619 14:37:43.924760 (+ 1338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:43.924764 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:43.924819 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:43.924926 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:43.926158 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:43.926873 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:43.928205 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:43.928412 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:43.928501 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:43.931047 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:43.931050 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:43.931052 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:43.931075 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:43.932346 (+ 1271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:43.932990 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:43.932993 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:43.932994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:43.933077 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:43.933421 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:43.933423 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:43.933424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:43.933614 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:43.933689 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:43.934159 (+ 470us) server_negotiation.cc:299] Negotiation successful
0619 14:37:43.934254 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 14:37:43.935173 4821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:43.922841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37313 (local address 127.0.98.2:59405)
0619 14:37:43.923264 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:43.923305 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:43.924527 (+ 1222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:43.925095 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:43.925102 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:43.925519 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:43.926019 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:43.926038 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:43.926987 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:43.926990 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:43.928090 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:43.928101 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:43.928562 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:43.928567 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:43.928773 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:43.929299 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:43.929314 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:43.930845 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:43.932453 (+ 1608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:43.932458 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:43.932474 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:43.932854 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:43.933166 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:43.933169 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:43.933171 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:43.933338 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:43.933708 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:43.933714 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:43.933966 (+ 252us) client_negotiation.cc:769] Sending connection context
0619 14:37:43.934123 (+ 157us) client_negotiation.cc:240] Negotiation successful
0619 14:37:43.934367 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":97,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 14:37:43.936229 4819 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37313
I20260619 14:37:43.936477 4819 heartbeater.cc:464] Registering TS with master...
I20260619 14:37:43.936996 4819 heartbeater.cc:511] Master 127.0.98.62:37313 requested a full tablet report, sending...
I20260619 14:37:43.938053 4496 ts_manager.cc:194] Registered new tserver with Master: 7a1e11915e054aa887d3a603b64a486d (127.0.98.2:33365)
I20260619 14:37:43.938755 4496 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59405
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 14:37:43.949816 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37313
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:37:44.056126 4824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:37:44.056370 4824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:37:44.056428 4824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:37:44.060302 4824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:37:44.060360 4824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:37:44.060438 4824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:37:44.065485 4824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:37:44.066478 4824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:37:44.067437 4824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:37:44.076479 4832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:44.076479 4830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:44.076483 4829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:44.076711 4824 server_base.cc:1034] running on GCE node
I20260619 14:37:44.077342 4824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:37:44.077936 4824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:37:44.079111 4824 hybrid_clock.cc:648] HybridClock initialized: now 1781879864079091 us; error 32 us; skew 500 ppm
Jun 19 14:37:44 dist-test-slave-32hj krb5kdc[4452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:37:44.081843 4824 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:37:44.082520 4824 fs_manager.cc:359] Metadata directory not provided
I20260619 14:37:44.082578 4824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:37:44.082754 4824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:37:44.084776 4824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/ts-2/data/instance:
uuid: "c137592f1c574c0f98e680d06d7ceef8"
format_stamp: "Formatted at 2026-06-19 14:37:44 on dist-test-slave-32hj"
server_key: "799987887a7aa16e475e73d1af01d421"
server_key_iv: "a3e47a19a3f664319a5993612fdf1f68"
server_key_version: "encryptionkey@0"
I20260619 14:37:44.085285 4824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c137592f1c574c0f98e680d06d7ceef8"
format_stamp: "Formatted at 2026-06-19 14:37:44 on dist-test-slave-32hj"
server_key: "799987887a7aa16e475e73d1af01d421"
server_key_iv: "a3e47a19a3f664319a5993612fdf1f68"
server_key_version: "encryptionkey@0"
I20260619 14:37:44.088689 4824 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 14:37:44.091008 4838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:44.092168 4824 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:37:44.092260 4824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/ts-2/wal
uuid: "c137592f1c574c0f98e680d06d7ceef8"
format_stamp: "Formatted at 2026-06-19 14:37:44 on dist-test-slave-32hj"
server_key: "799987887a7aa16e475e73d1af01d421"
server_key_iv: "a3e47a19a3f664319a5993612fdf1f68"
server_key_version: "encryptionkey@0"
I20260619 14:37:44.092360 4824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:37:44.115605 4824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:37:44.118785 4824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:37:44.118954 4824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:37:44.119529 4824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:37:44.120332 4824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:37:44.120378 4824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:44.120415 4824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:37:44.120441 4824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:44.128571 4824 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34537
I20260619 14:37:44.128600 4951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34537 every 8 connection(s)
I20260619 14:37:44.129136 4824 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:37:44.135174 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 4824
I20260619 14:37:44.135258 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:37:44.135501 392 external_mini_cluster.cc:1428] Setting key 53b3ada250508b446d7459fb852bfe0b
Jun 19 14:37:44 dist-test-slave-32hj krb5kdc[4452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:37:44.141786 4535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:44.130859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51941 (local address 127.0.98.62:37313)
0619 14:37:44.130989 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:44.130993 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:44.131826 (+ 833us) server_negotiation.cc:407] Connection header received
0619 14:37:44.133170 (+ 1344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:44.133174 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:44.133228 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:44.133301 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:44.134398 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:44.134926 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:44.136183 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:44.136459 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:44.136547 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:44.138810 (+ 2263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:44.138814 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:44.138817 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:44.138840 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:44.139960 (+ 1120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:44.140399 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:44.140403 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:44.140405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:44.140479 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:44.140789 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:44.140792 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:44.140794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:44.140963 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:44.141046 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:44.141509 (+ 463us) server_negotiation.cc:299] Negotiation successful
0619 14:37:44.141624 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 14:37:44.142495 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:44.131194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37313 (local address 127.0.98.3:51941)
0619 14:37:44.131653 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:44.131694 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:44.132976 (+ 1282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:44.133456 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:44.133463 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:44.133797 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:44.134255 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:44.134279 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:44.135051 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:44.135054 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:44.136072 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:44.136082 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:44.136591 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:44.136595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:44.136810 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:44.137363 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:44.137381 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:44.138656 (+ 1275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:44.140057 (+ 1401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:44.140062 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:44.140077 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:44.140313 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:44.140554 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:44.140557 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:44.140559 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:44.140681 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:44.141053 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:44.141059 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:44.141289 (+ 230us) client_negotiation.cc:769] Sending connection context
0619 14:37:44.141437 (+ 148us) client_negotiation.cc:240] Negotiation successful
0619 14:37:44.141726 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":107,"threads_started":1}
I20260619 14:37:44.143424 4952 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37313
I20260619 14:37:44.143637 4952 heartbeater.cc:464] Registering TS with master...
I20260619 14:37:44.144057 4952 heartbeater.cc:511] Master 127.0.98.62:37313 requested a full tablet report, sending...
I20260619 14:37:44.144970 4496 ts_manager.cc:194] Registered new tserver with Master: c137592f1c574c0f98e680d06d7ceef8 (127.0.98.3:34537)
I20260619 14:37:44.145490 4496 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51941
I20260619 14:37:44.148934 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:37:44.156543 4535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:44.150543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39908 (local address 127.0.98.62:37313)
0619 14:37:44.150646 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:44.150650 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:44.150771 (+ 121us) server_negotiation.cc:407] Connection header received
0619 14:37:44.150862 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:44.150864 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:44.150899 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:44.150958 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:44.151523 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:44.152013 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:44.152857 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:44.153058 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:44.153131 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:44.154252 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:44.154255 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:44.154257 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:44.154277 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:44.155346 (+ 1069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:44.155691 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:44.155694 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:44.155695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:44.155757 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:44.155931 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:44.155933 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:44.155934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:44.156108 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:44.156201 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:44.156345 (+ 144us) server_negotiation.cc:299] Negotiation successful
0619 14:37:44.156425 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 14:37:44.158905 4496 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39908:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:37:44.161736 4496 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:37:44.175524 4964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:44.169867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36831 (local address 127.0.0.1:58102)
0619 14:37:44.170972 (+ 1105us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:44.170982 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:44.171065 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:44.171496 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:44.171498 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:44.171516 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:37:44.171598 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:44.171603 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:44.172774 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:44.172778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:44.174033 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:44.174043 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:44.175128 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:44.175132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:44.175215 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:44.175237 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 14:37:44.175310 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 14:37:44.175380 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1057,"thread_start_us":84,"threads_started":1}
I20260619 14:37:44.175571 4967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:44.170265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58102 (local address 127.0.98.1:36831)
0619 14:37:44.170853 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:44.170862 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:44.171063 (+ 201us) server_negotiation.cc:407] Connection header received
0619 14:37:44.171168 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:44.171173 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:44.171364 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:44.171628 (+ 264us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:37:44.171785 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:44.172627 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:44.174152 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:44.175026 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:44.175119 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:44.175255 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:44.175339 (+ 84us) server_negotiation.cc:299] Negotiation successful
0619 14:37:44.175429 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":86,"threads_started":1}
I20260619 14:37:44.175752 4968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:44.170291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50864 (local address 127.0.98.2:33365)
0619 14:37:44.170772 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:44.170784 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:44.170806 (+ 22us) server_negotiation.cc:407] Connection header received
0619 14:37:44.170871 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:44.170875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:44.171029 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:44.171160 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:37:44.171429 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:44.172429 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:44.173966 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:44.174721 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:44.174831 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:44.174989 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:44.175460 (+ 471us) server_negotiation.cc:299] Negotiation successful
0619 14:37:44.175563 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":103,"threads_started":1}
I20260619 14:37:44.175364 4966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:44.169649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33365 (local address 127.0.0.1:50864)
0619 14:37:44.170574 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:44.170588 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:44.170670 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:44.171166 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:44.171172 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:44.171194 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:37:44.171326 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:44.171333 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:44.172584 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:44.172587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:44.173849 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:44.173858 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:44.174832 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:44.174836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:44.174942 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:44.174995 (+ 53us) client_negotiation.cc:769] Sending connection context
0619 14:37:44.175103 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 14:37:44.175217 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":866,"mutex_wait_us":49,"thread_start_us":99,"threads_started":1}
I20260619 14:37:44.175966 4963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:44.169507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34537 (local address 127.0.0.1:57684)
0619 14:37:44.170376 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:44.170429 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:44.170547 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:44.171825 (+ 1278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:44.171828 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:44.171845 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:37:44.171920 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:44.171926 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:44.173262 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:44.173266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:44.174295 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:44.174302 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:44.175364 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:44.175367 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:44.175463 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:44.175484 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 14:37:44.175548 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 14:37:44.175607 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":70,"threads_started":1}
I20260619 14:37:44.176358 4965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:44.169649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57684 (local address 127.0.98.3:34537)
0619 14:37:44.171289 (+ 1640us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:44.171299 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:44.171317 (+ 18us) server_negotiation.cc:407] Connection header received
0619 14:37:44.171380 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:44.171385 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:44.171561 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:44.171683 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:37:44.172041 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:44.173119 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:44.174393 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:44.175097 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:44.175182 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:44.175308 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:44.176118 (+ 810us) server_negotiation.cc:299] Negotiation successful
0619 14:37:44.176216 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1549,"thread_start_us":142,"threads_started":1}
I20260619 14:37:44.178516 4753 tablet_service.cc:1467] Processing CreateTablet for tablet b043cb64251e4eafb9bad53ac3a9db24 (DEFAULT_TABLE table=test-table [id=12407925e7ba41e6a537cc4f653519d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:37:44.178664 4886 tablet_service.cc:1467] Processing CreateTablet for tablet b043cb64251e4eafb9bad53ac3a9db24 (DEFAULT_TABLE table=test-table [id=12407925e7ba41e6a537cc4f653519d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:37:44.178664 4620 tablet_service.cc:1467] Processing CreateTablet for tablet b043cb64251e4eafb9bad53ac3a9db24 (DEFAULT_TABLE table=test-table [id=12407925e7ba41e6a537cc4f653519d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:37:44.179384 4753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b043cb64251e4eafb9bad53ac3a9db24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:44.179634 4620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b043cb64251e4eafb9bad53ac3a9db24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:44.179639 4886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b043cb64251e4eafb9bad53ac3a9db24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:44.185977 4969 tablet_bootstrap.cc:492] T b043cb64251e4eafb9bad53ac3a9db24 P 7a1e11915e054aa887d3a603b64a486d: Bootstrap starting.
I20260619 14:37:44.186880 4970 tablet_bootstrap.cc:492] T b043cb64251e4eafb9bad53ac3a9db24 P c137592f1c574c0f98e680d06d7ceef8: Bootstrap starting.
I20260619 14:37:44.188086 4969 tablet_bootstrap.cc:654] T b043cb64251e4eafb9bad53ac3a9db24 P 7a1e11915e054aa887d3a603b64a486d: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:44.188547 4970 tablet_bootstrap.cc:654] T b043cb64251e4eafb9bad53ac3a9db24 P c137592f1c574c0f98e680d06d7ceef8: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:44.189038 4971 tablet_bootstrap.cc:492] T b043cb64251e4eafb9bad53ac3a9db24 P f4605c31c89d4b219a87d36179c22387: Bootstrap starting.
I20260619 14:37:44.189158 4969 log.cc:824] T b043cb64251e4eafb9bad53ac3a9db24 P 7a1e11915e054aa887d3a603b64a486d: Log is configured to *not* fsync() on all Append() calls
I20260619 14:37:44.189709 4970 log.cc:824] T b043cb64251e4eafb9bad53ac3a9db24 P c137592f1c574c0f98e680d06d7ceef8: Log is configured to *not* fsync() on all Append() calls
I20260619 14:37:44.190973 4971 tablet_bootstrap.cc:654] T b043cb64251e4eafb9bad53ac3a9db24 P f4605c31c89d4b219a87d36179c22387: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:44.191083 4969 tablet_bootstrap.cc:492] T b043cb64251e4eafb9bad53ac3a9db24 P 7a1e11915e054aa887d3a603b64a486d: No bootstrap required, opened a new log
I20260619 14:37:44.191267 4969 ts_tablet_manager.cc:1397] T b043cb64251e4eafb9bad53ac3a9db24 P 7a1e11915e054aa887d3a603b64a486d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:37:44.191601 4970 tablet_bootstrap.cc:492] T b043cb64251e4eafb9bad53ac3a9db24 P c137592f1c574c0f98e680d06d7ceef8: No bootstrap required, opened a new log
I20260619 14:37:44.191780 4970 ts_tablet_manager.cc:1397] T b043cb64251e4eafb9bad53ac3a9db24 P c137592f1c574c0f98e680d06d7ceef8: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 14:37:44.192107 4971 log.cc:824] T b043cb64251e4eafb9bad53ac3a9db24 P f4605c31c89d4b219a87d36179c22387: Log is configured to *not* fsync() on all Append() calls
I20260619 14:37:44.194041 4971 tablet_bootstrap.cc:492] T b043cb64251e4eafb9bad53ac3a9db24 P f4605c31c89d4b219a87d36179c22387: No bootstrap required, opened a new log
I20260619 14:37:44.194221 4971 ts_tablet_manager.cc:1397] T b043cb64251e4eafb9bad53ac3a9db24 P f4605c31c89d4b219a87d36179c22387: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:37:44.194763 4969 raft_consensus.cc:348] T b043cb64251e4eafb9bad53ac3a9db24 P 7a1e11915e054aa887d3a603b64a486d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c137592f1c574c0f98e680d06d7ceef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34537 } } peers { permanent_uuid: "7a1e11915e054aa887d3a603b64a486d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33365 } } peers { permanent_uuid: "f4605c31c89d4b219a87d36179c22387" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36831 } }
I20260619 14:37:44.195036 4969 raft_consensus.cc:374] T b043cb64251e4eafb9bad53ac3a9db24 P 7a1e11915e054aa887d3a603b64a486d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:44.195097 4969 raft_consensus.cc:729] T b043cb64251e4eafb9bad53ac3a9db24 P 7a1e11915e054aa887d3a603b64a486d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a1e11915e054aa887d3a603b64a486d, State: Initialized, Role: FOLLOWER
I20260619 14:37:44.195343 4970 raft_consensus.cc:348] T b043cb64251e4eafb9bad53ac3a9db24 P c137592f1c574c0f98e680d06d7ceef8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c137592f1c574c0f98e680d06d7ceef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34537 } } peers { permanent_uuid: "7a1e11915e054aa887d3a603b64a486d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33365 } } peers { permanent_uuid: "f4605c31c89d4b219a87d36179c22387" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36831 } }
I20260619 14:37:44.195494 4969 consensus_queue.cc:260] T b043cb64251e4eafb9bad53ac3a9db24 P 7a1e11915e054aa887d3a603b64a486d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c137592f1c574c0f98e680d06d7ceef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34537 } } peers { permanent_uuid: "7a1e11915e054aa887d3a603b64a486d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33365 } } peers { permanent_uuid: "f4605c31c89d4b219a87d36179c22387" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36831 } }
I20260619 14:37:44.195595 4970 raft_consensus.cc:374] T b043cb64251e4eafb9bad53ac3a9db24 P c137592f1c574c0f98e680d06d7ceef8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:44.195644 4970 raft_consensus.cc:729] T b043cb64251e4eafb9bad53ac3a9db24 P c137592f1c574c0f98e680d06d7ceef8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c137592f1c574c0f98e680d06d7ceef8, State: Initialized, Role: FOLLOWER
I20260619 14:37:44.196035 4970 consensus_queue.cc:260] T b043cb64251e4eafb9bad53ac3a9db24 P c137592f1c574c0f98e680d06d7ceef8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c137592f1c574c0f98e680d06d7ceef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34537 } } peers { permanent_uuid: "7a1e11915e054aa887d3a603b64a486d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33365 } } peers { permanent_uuid: "f4605c31c89d4b219a87d36179c22387" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36831 } }
I20260619 14:37:44.196332 4969 ts_tablet_manager.cc:1428] T b043cb64251e4eafb9bad53ac3a9db24 P 7a1e11915e054aa887d3a603b64a486d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 14:37:44.196668 4819 heartbeater.cc:503] Master 127.0.98.62:37313 was elected leader, sending a full tablet report...
I20260619 14:37:44.196668 4952 heartbeater.cc:503] Master 127.0.98.62:37313 was elected leader, sending a full tablet report...
I20260619 14:37:44.196982 4970 ts_tablet_manager.cc:1428] T b043cb64251e4eafb9bad53ac3a9db24 P c137592f1c574c0f98e680d06d7ceef8: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 14:37:44.197980 4971 raft_consensus.cc:348] T b043cb64251e4eafb9bad53ac3a9db24 P f4605c31c89d4b219a87d36179c22387 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c137592f1c574c0f98e680d06d7ceef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34537 } } peers { permanent_uuid: "7a1e11915e054aa887d3a603b64a486d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33365 } } peers { permanent_uuid: "f4605c31c89d4b219a87d36179c22387" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36831 } }
I20260619 14:37:44.198213 4971 raft_consensus.cc:374] T b043cb64251e4eafb9bad53ac3a9db24 P f4605c31c89d4b219a87d36179c22387 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:44.198262 4971 raft_consensus.cc:729] T b043cb64251e4eafb9bad53ac3a9db24 P f4605c31c89d4b219a87d36179c22387 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4605c31c89d4b219a87d36179c22387, State: Initialized, Role: FOLLOWER
I20260619 14:37:44.198630 4971 consensus_queue.cc:260] T b043cb64251e4eafb9bad53ac3a9db24 P f4605c31c89d4b219a87d36179c22387 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c137592f1c574c0f98e680d06d7ceef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34537 } } peers { permanent_uuid: "7a1e11915e054aa887d3a603b64a486d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33365 } } peers { permanent_uuid: "f4605c31c89d4b219a87d36179c22387" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36831 } }
I20260619 14:37:44.199321 4686 heartbeater.cc:503] Master 127.0.98.62:37313 was elected leader, sending a full tablet report...
I20260619 14:37:44.199448 4971 ts_tablet_manager.cc:1428] T b043cb64251e4eafb9bad53ac3a9db24 P f4605c31c89d4b219a87d36179c22387: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260619 14:37:44.224992 4687 tablet.cc:2369] T b043cb64251e4eafb9bad53ac3a9db24 P f4605c31c89d4b219a87d36179c22387: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:37:44.287873 4977 raft_consensus.cc:482] T b043cb64251e4eafb9bad53ac3a9db24 P f4605c31c89d4b219a87d36179c22387 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:37:44.288103 4977 raft_consensus.cc:504] T b043cb64251e4eafb9bad53ac3a9db24 P f4605c31c89d4b219a87d36179c22387 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c137592f1c574c0f98e680d06d7ceef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34537 } } peers { permanent_uuid: "7a1e11915e054aa887d3a603b64a486d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33365 } } peers { permanent_uuid: "f4605c31c89d4b219a87d36179c22387" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36831 } }
I20260619 14:37:44.289309 4977 leader_election.cc:288] T b043cb64251e4eafb9bad53ac3a9db24 P f4605c31c89d4b219a87d36179c22387 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c137592f1c574c0f98e680d06d7ceef8 (127.0.98.3:34537), 7a1e11915e054aa887d3a603b64a486d (127.0.98.2:33365)
I20260619 14:37:44.294575 4968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:44.289728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35221 (local address 127.0.98.2:33365)
0619 14:37:44.289836 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:44.289840 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:44.289913 (+ 73us) server_negotiation.cc:407] Connection header received
0619 14:37:44.290065 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:44.290068 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:44.290120 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:44.290198 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:37:44.290539 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:44.291318 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:44.293216 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:44.293847 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:44.293971 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:44.294034 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:44.294369 (+ 335us) server_negotiation.cc:299] Negotiation successful
0619 14:37:44.294448 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 14:37:44.294574 4965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:44.289601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58983 (local address 127.0.98.3:34537)
0619 14:37:44.289717 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:44.289721 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:44.289924 (+ 203us) server_negotiation.cc:407] Connection header received
0619 14:37:44.290065 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:44.290068 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:44.290121 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:44.290198 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:37:44.290539 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:44.291318 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:44.293216 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:44.293895 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:44.293986 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:44.294036 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:44.294369 (+ 333us) server_negotiation.cc:299] Negotiation successful
0619 14:37:44.294449 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 14:37:44.294605 4979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:44.289469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34537 (local address 127.0.98.1:58983)
0619 14:37:44.289823 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:44.289838 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:44.289963 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:44.290215 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:44.290218 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:44.290241 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:37:44.290438 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:44.290444 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:44.291435 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:44.291440 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:44.293081 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:44.293093 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:44.294016 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:44.294020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:44.294229 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:44.294273 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 14:37:44.294332 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 14:37:44.294453 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":251,"spinlock_wait_cycles":21504,"thread_start_us":109,"threads_started":1}
I20260619 14:37:44.294615 4978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:44.289530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33365 (local address 127.0.98.1:35221)
0619 14:37:44.289829 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:44.289840 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:44.289963 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:44.290215 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:44.290218 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:44.290241 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:37:44.290438 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:44.290444 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:44.291437 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:44.291440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:44.293081 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:44.293093 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:44.294016 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:44.294020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:44.294213 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:44.294256 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 14:37:44.294338 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 14:37:44.294453 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":270,"spinlock_wait_cycles":27392,"thread_start_us":97,"threads_started":1}
I20260619 14:37:44.295284 4773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b043cb64251e4eafb9bad53ac3a9db24" candidate_uuid: "f4605c31c89d4b219a87d36179c22387" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a1e11915e054aa887d3a603b64a486d" is_pre_election: true
I20260619 14:37:44.295327 4906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b043cb64251e4eafb9bad53ac3a9db24" candidate_uuid: "f4605c31c89d4b219a87d36179c22387" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c137592f1c574c0f98e680d06d7ceef8" is_pre_election: true
I20260619 14:37:44.295555 4773 raft_consensus.cc:2454] T b043cb64251e4eafb9bad53ac3a9db24 P 7a1e11915e054aa887d3a603b64a486d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4605c31c89d4b219a87d36179c22387 in term 0.
I20260619 14:37:44.295581 4906 raft_consensus.cc:2454] T b043cb64251e4eafb9bad53ac3a9db24 P c137592f1c574c0f98e680d06d7ceef8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4605c31c89d4b219a87d36179c22387 in term 0.
I20260619 14:37:44.295989 4574 leader_election.cc:302] T b043cb64251e4eafb9bad53ac3a9db24 P f4605c31c89d4b219a87d36179c22387 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7a1e11915e054aa887d3a603b64a486d, c137592f1c574c0f98e680d06d7ceef8, f4605c31c89d4b219a87d36179c22387; no voters:
I20260619 14:37:44.296247 4977 raft_consensus.cc:2781] T b043cb64251e4eafb9bad53ac3a9db24 P f4605c31c89d4b219a87d36179c22387 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:37:44.296350 4977 raft_consensus.cc:482] T b043cb64251e4eafb9bad53ac3a9db24 P f4605c31c89d4b219a87d36179c22387 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:37:44.296399 4977 raft_consensus.cc:3037] T b043cb64251e4eafb9bad53ac3a9db24 P f4605c31c89d4b219a87d36179c22387 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:44.297366 4977 raft_consensus.cc:504] T b043cb64251e4eafb9bad53ac3a9db24 P f4605c31c89d4b219a87d36179c22387 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c137592f1c574c0f98e680d06d7ceef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34537 } } peers { permanent_uuid: "7a1e11915e054aa887d3a603b64a486d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33365 } } peers { permanent_uuid: "f4605c31c89d4b219a87d36179c22387" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36831 } }
I20260619 14:37:44.297631 4977 leader_election.cc:288] T b043cb64251e4eafb9bad53ac3a9db24 P f4605c31c89d4b219a87d36179c22387 [CANDIDATE]: Term 1 election: Requested vote from peers c137592f1c574c0f98e680d06d7ceef8 (127.0.98.3:34537), 7a1e11915e054aa887d3a603b64a486d (127.0.98.2:33365)
I20260619 14:37:44.297928 4773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b043cb64251e4eafb9bad53ac3a9db24" candidate_uuid: "f4605c31c89d4b219a87d36179c22387" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a1e11915e054aa887d3a603b64a486d"
I20260619 14:37:44.297924 4906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b043cb64251e4eafb9bad53ac3a9db24" candidate_uuid: "f4605c31c89d4b219a87d36179c22387" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c137592f1c574c0f98e680d06d7ceef8"
I20260619 14:37:44.298050 4773 raft_consensus.cc:3037] T b043cb64251e4eafb9bad53ac3a9db24 P 7a1e11915e054aa887d3a603b64a486d [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:44.298050 4906 raft_consensus.cc:3037] T b043cb64251e4eafb9bad53ac3a9db24 P c137592f1c574c0f98e680d06d7ceef8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:44.299268 4906 raft_consensus.cc:2454] T b043cb64251e4eafb9bad53ac3a9db24 P c137592f1c574c0f98e680d06d7ceef8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4605c31c89d4b219a87d36179c22387 in term 1.
I20260619 14:37:44.299346 4773 raft_consensus.cc:2454] T b043cb64251e4eafb9bad53ac3a9db24 P 7a1e11915e054aa887d3a603b64a486d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4605c31c89d4b219a87d36179c22387 in term 1.
I20260619 14:37:44.299569 4574 leader_election.cc:302] T b043cb64251e4eafb9bad53ac3a9db24 P f4605c31c89d4b219a87d36179c22387 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a1e11915e054aa887d3a603b64a486d, f4605c31c89d4b219a87d36179c22387; no voters:
I20260619 14:37:44.299731 4977 raft_consensus.cc:2781] T b043cb64251e4eafb9bad53ac3a9db24 P f4605c31c89d4b219a87d36179c22387 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:37:44.299932 4977 raft_consensus.cc:686] T b043cb64251e4eafb9bad53ac3a9db24 P f4605c31c89d4b219a87d36179c22387 [term 1 LEADER]: Becoming Leader. State: Replica: f4605c31c89d4b219a87d36179c22387, State: Running, Role: LEADER
I20260619 14:37:44.300251 4977 consensus_queue.cc:237] T b043cb64251e4eafb9bad53ac3a9db24 P f4605c31c89d4b219a87d36179c22387 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c137592f1c574c0f98e680d06d7ceef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34537 } } peers { permanent_uuid: "7a1e11915e054aa887d3a603b64a486d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33365 } } peers { permanent_uuid: "f4605c31c89d4b219a87d36179c22387" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36831 } }
I20260619 14:37:44.303182 4495 catalog_manager.cc:5495] T b043cb64251e4eafb9bad53ac3a9db24 P f4605c31c89d4b219a87d36179c22387 reported cstate change: term changed from 0 to 1, leader changed from <none> to f4605c31c89d4b219a87d36179c22387 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "f4605c31c89d4b219a87d36179c22387" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c137592f1c574c0f98e680d06d7ceef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a1e11915e054aa887d3a603b64a486d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4605c31c89d4b219a87d36179c22387" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36831 } health_report { overall_health: HEALTHY } } }
I20260619 14:37:44.324700 4967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:44.321569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58110 (local address 127.0.98.1:36831)
0619 14:37:44.321663 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:44.321667 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:44.321712 (+ 45us) server_negotiation.cc:407] Connection header received
0619 14:37:44.321863 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:44.321865 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:44.321919 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:44.321994 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:37:44.322156 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:44.322618 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:44.323427 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:44.323639 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:44.323717 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:44.323968 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:37:44.324381 (+ 413us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:37:44.324484 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:44.324553 (+ 69us) server_negotiation.cc:299] Negotiation successful
0619 14:37:44.324601 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 14:37:44.330281 4773 raft_consensus.cc:1261] T b043cb64251e4eafb9bad53ac3a9db24 P 7a1e11915e054aa887d3a603b64a486d [term 1 FOLLOWER]: Refusing update from remote peer f4605c31c89d4b219a87d36179c22387: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:37:44.330281 4906 raft_consensus.cc:1261] T b043cb64251e4eafb9bad53ac3a9db24 P c137592f1c574c0f98e680d06d7ceef8 [term 1 FOLLOWER]: Refusing update from remote peer f4605c31c89d4b219a87d36179c22387: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:37:44.330902 4977 consensus_queue.cc:1035] T b043cb64251e4eafb9bad53ac3a9db24 P f4605c31c89d4b219a87d36179c22387 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c137592f1c574c0f98e680d06d7ceef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:37:44.331094 4980 consensus_queue.cc:1035] T b043cb64251e4eafb9bad53ac3a9db24 P f4605c31c89d4b219a87d36179c22387 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a1e11915e054aa887d3a603b64a486d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:37:44.337884 4985 mvcc.cc:205] Tried to move back new op lower bound from 7298579924289974272 to 7298579924176453632. Current Snapshot: MvccSnapshot[applied={T|T < 7298579924289974272}]
I20260619 14:37:44.344861 4496 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39908:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:37:44.344995 4496 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39908:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:37:44.347124 4496 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a227e022210d46c6b8f11837b3b09f0e: Sending DeleteTablet for 3 replicas of tablet b043cb64251e4eafb9bad53ac3a9db24
I20260619 14:37:44.347777 4886 tablet_service.cc:1514] Processing DeleteTablet for tablet b043cb64251e4eafb9bad53ac3a9db24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:37:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57684
I20260619 14:37:44.347821 4753 tablet_service.cc:1514] Processing DeleteTablet for tablet b043cb64251e4eafb9bad53ac3a9db24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:37:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50864
I20260619 14:37:44.348387 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 4558
I20260619 14:37:44.348954 4993 tablet_replica.cc:330] T b043cb64251e4eafb9bad53ac3a9db24 P c137592f1c574c0f98e680d06d7ceef8: stopping tablet replica
I20260619 14:37:44.348975 4992 tablet_replica.cc:330] T b043cb64251e4eafb9bad53ac3a9db24 P 7a1e11915e054aa887d3a603b64a486d: stopping tablet replica
I20260619 14:37:44.349259 4993 raft_consensus.cc:2229] T b043cb64251e4eafb9bad53ac3a9db24 P c137592f1c574c0f98e680d06d7ceef8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:37:44.349249 4992 raft_consensus.cc:2229] T b043cb64251e4eafb9bad53ac3a9db24 P 7a1e11915e054aa887d3a603b64a486d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:37:44.349546 4993 raft_consensus.cc:2258] T b043cb64251e4eafb9bad53ac3a9db24 P c137592f1c574c0f98e680d06d7ceef8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:37:44.349546 4992 raft_consensus.cc:2258] T b043cb64251e4eafb9bad53ac3a9db24 P 7a1e11915e054aa887d3a603b64a486d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:37:44.350960 4992 ts_tablet_manager.cc:1905] T b043cb64251e4eafb9bad53ac3a9db24 P 7a1e11915e054aa887d3a603b64a486d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:37:44.350952 4993 ts_tablet_manager.cc:1905] T b043cb64251e4eafb9bad53ac3a9db24 P c137592f1c574c0f98e680d06d7ceef8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:37:44.353506 4992 ts_tablet_manager.cc:1918] T b043cb64251e4eafb9bad53ac3a9db24 P 7a1e11915e054aa887d3a603b64a486d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:37:44.353506 4993 ts_tablet_manager.cc:1918] T b043cb64251e4eafb9bad53ac3a9db24 P c137592f1c574c0f98e680d06d7ceef8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:37:44.353605 4993 log.cc:1196] T b043cb64251e4eafb9bad53ac3a9db24 P c137592f1c574c0f98e680d06d7ceef8: Deleting WAL directory at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/ts-2/wal/wals/b043cb64251e4eafb9bad53ac3a9db24
I20260619 14:37:44.353605 4992 log.cc:1196] T b043cb64251e4eafb9bad53ac3a9db24 P 7a1e11915e054aa887d3a603b64a486d: Deleting WAL directory at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781879837648036-392-0/minicluster-data/ts-1/wal/wals/b043cb64251e4eafb9bad53ac3a9db24
I20260619 14:37:44.353915 4993 ts_tablet_manager.cc:1939] T b043cb64251e4eafb9bad53ac3a9db24 P c137592f1c574c0f98e680d06d7ceef8: Deleting consensus metadata
I20260619 14:37:44.353915 4992 ts_tablet_manager.cc:1939] T b043cb64251e4eafb9bad53ac3a9db24 P 7a1e11915e054aa887d3a603b64a486d: Deleting consensus metadata
W20260619 14:37:44.354622 4482 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36831 (error 108)
I20260619 14:37:44.355077 4481 catalog_manager.cc:4841] TS 7a1e11915e054aa887d3a603b64a486d (127.0.98.2:33365): tablet b043cb64251e4eafb9bad53ac3a9db24 (table test-table [id=12407925e7ba41e6a537cc4f653519d9]) successfully deleted
I20260619 14:37:44.355077 4480 catalog_manager.cc:4841] TS c137592f1c574c0f98e680d06d7ceef8 (127.0.98.3:34537): tablet b043cb64251e4eafb9bad53ac3a9db24 (table test-table [id=12407925e7ba41e6a537cc4f653519d9]) successfully deleted
I20260619 14:37:44.355530 4966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:44.355260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36831 (local address 127.0.0.1:58122)
0619 14:37:44.355367 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:44.355427 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260619 14:37:44.355574 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 4691
W20260619 14:37:44.355676 4482 catalog_manager.cc:4584] TS f4605c31c89d4b219a87d36179c22387 (127.0.98.1:36831): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b043cb64251e4eafb9bad53ac3a9db24: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36831: connect: Connection refused (error 111)
I20260619 14:37:44.360273 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 4824
I20260619 14:37:44.364450 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 4465
2026-06-19T14:37:44Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2144 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-19T14:37:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:37:44Z Disabled control of system clock
I20260619 14:37:44.391180 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36715
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36715
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 14:37:44.495025 4998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:37:44.495275 4998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:37:44.495309 4998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:37:44.499212 4998 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:37:44.499266 4998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:37:44.499280 4998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:37:44.499301 4998 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:37:44.499315 4998 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:37:44.504002 4998 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36715
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:37:44.504983 4998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:37:44.506310 4998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:37:44.514266 5003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:44.514376 4998 server_base.cc:1034] running on GCE node
W20260619 14:37:44.514281 5006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:44.514288 5004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:44.515141 4998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:37:44.516181 4998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:37:44.517339 4998 hybrid_clock.cc:648] HybridClock initialized: now 1781879864517311 us; error 40 us; skew 500 ppm
I20260619 14:37:44.518987 4998 webserver.cc:460] Webserver started at http://127.0.98.62:41391/ using document root <none> and password file <none>
I20260619 14:37:44.519488 4998 fs_manager.cc:359] Metadata directory not provided
I20260619 14:37:44.519542 4998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:37:44.519693 4998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:37:44.521646 4998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/master-0/data/instance:
uuid: "3b3447ac5b3343878b45d540f9576301"
format_stamp: "Formatted at 2026-06-19 14:37:44 on dist-test-slave-32hj"
server_key: "674e9e57691145f9bfc28fe6fcef9d13"
server_key_iv: "c4583710a738a2de8877fe5b1e205e35"
server_key_version: "encryptionkey@0"
I20260619 14:37:44.522104 4998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/master-0/wal/instance:
uuid: "3b3447ac5b3343878b45d540f9576301"
format_stamp: "Formatted at 2026-06-19 14:37:44 on dist-test-slave-32hj"
server_key: "674e9e57691145f9bfc28fe6fcef9d13"
server_key_iv: "c4583710a738a2de8877fe5b1e205e35"
server_key_version: "encryptionkey@0"
I20260619 14:37:44.525378 4998 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:37:44.527581 5012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:44.528697 4998 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:37:44.528841 4998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/master-0/wal
uuid: "3b3447ac5b3343878b45d540f9576301"
format_stamp: "Formatted at 2026-06-19 14:37:44 on dist-test-slave-32hj"
server_key: "674e9e57691145f9bfc28fe6fcef9d13"
server_key_iv: "c4583710a738a2de8877fe5b1e205e35"
server_key_version: "encryptionkey@0"
I20260619 14:37:44.528936 4998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:37:44.533239 4998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:37:44.533780 4998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:37:44.533923 4998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:37:44.540027 4998 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36715
I20260619 14:37:44.540055 5064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36715 every 8 connection(s)
I20260619 14:37:44.540921 4998 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:37:44.543779 5065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:44.545806 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 4998
I20260619 14:37:44.545897 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/master-0/wal/instance
I20260619 14:37:44.546149 392 external_mini_cluster.cc:1428] Setting key 4d64b47d433b6fd395e8a5ccd6c5b739
I20260619 14:37:44.549217 5065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b3447ac5b3343878b45d540f9576301: Bootstrap starting.
I20260619 14:37:44.551234 5068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:44.547467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53296 (local address 127.0.98.62:36715)
0619 14:37:44.548070 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:44.548083 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:44.548117 (+ 34us) server_negotiation.cc:407] Connection header received
0619 14:37:44.548703 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:44.548748 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:44.549092 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:44.549383 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:44.550101 (+ 718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:44.550105 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:44.550116 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:37:44.550125 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:44.550177 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:37:44.550192 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:44.550312 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:44.550378 (+ 66us) server_negotiation.cc:299] Negotiation successful
0619 14:37:44.550615 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":134,"threads_started":1}
I20260619 14:37:44.551401 5065 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b3447ac5b3343878b45d540f9576301: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:44.552060 5065 log.cc:824] T 00000000000000000000000000000000 P 3b3447ac5b3343878b45d540f9576301: Log is configured to *not* fsync() on all Append() calls
I20260619 14:37:44.553766 5065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b3447ac5b3343878b45d540f9576301: No bootstrap required, opened a new log
I20260619 14:37:44.557158 5065 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b3447ac5b3343878b45d540f9576301 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b3447ac5b3343878b45d540f9576301" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36715 } }
I20260619 14:37:44.557367 5065 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b3447ac5b3343878b45d540f9576301 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:44.557430 5065 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b3447ac5b3343878b45d540f9576301 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b3447ac5b3343878b45d540f9576301, State: Initialized, Role: FOLLOWER
I20260619 14:37:44.557808 5065 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b3447ac5b3343878b45d540f9576301 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b3447ac5b3343878b45d540f9576301" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36715 } }
I20260619 14:37:44.557929 5065 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b3447ac5b3343878b45d540f9576301 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:37:44.557996 5065 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b3447ac5b3343878b45d540f9576301 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:37:44.558089 5065 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b3447ac5b3343878b45d540f9576301 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:44.559019 5065 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b3447ac5b3343878b45d540f9576301 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b3447ac5b3343878b45d540f9576301" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36715 } }
I20260619 14:37:44.559316 5065 leader_election.cc:302] T 00000000000000000000000000000000 P 3b3447ac5b3343878b45d540f9576301 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b3447ac5b3343878b45d540f9576301; no voters:
I20260619 14:37:44.559571 5065 leader_election.cc:288] T 00000000000000000000000000000000 P 3b3447ac5b3343878b45d540f9576301 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:37:44.559649 5070 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b3447ac5b3343878b45d540f9576301 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:37:44.559877 5070 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b3447ac5b3343878b45d540f9576301 [term 1 LEADER]: Becoming Leader. State: Replica: 3b3447ac5b3343878b45d540f9576301, State: Running, Role: LEADER
I20260619 14:37:44.560199 5070 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b3447ac5b3343878b45d540f9576301 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b3447ac5b3343878b45d540f9576301" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36715 } }
I20260619 14:37:44.560614 5065 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b3447ac5b3343878b45d540f9576301 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:37:44.561655 5071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b3447ac5b3343878b45d540f9576301 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b3447ac5b3343878b45d540f9576301" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b3447ac5b3343878b45d540f9576301" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36715 } } }
I20260619 14:37:44.561710 5072 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b3447ac5b3343878b45d540f9576301 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b3447ac5b3343878b45d540f9576301. Latest consensus state: current_term: 1 leader_uuid: "3b3447ac5b3343878b45d540f9576301" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b3447ac5b3343878b45d540f9576301" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36715 } } }
I20260619 14:37:44.561774 5071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b3447ac5b3343878b45d540f9576301 [sys.catalog]: This master's current role is: LEADER
I20260619 14:37:44.561798 5072 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b3447ac5b3343878b45d540f9576301 [sys.catalog]: This master's current role is: LEADER
I20260619 14:37:44.562811 5079 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:37:44.565248 5079 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:37:44.570199 5079 catalog_manager.cc:1347] Generated new cluster ID: b2ddf27cf386463b81044e7a60222ed3
I20260619 14:37:44.570264 5079 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:37:44.587569 5079 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:37:44.587893 5079 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:37:44.593133 5079 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b3447ac5b3343878b45d540f9576301: Generated new TSK 0
I20260619 14:37:44.593776 5079 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:37:44.597179 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36715
--builtin_ntp_servers=127.0.98.20:35815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 14:37:44.703965 5089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:37:44.704195 5089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:37:44.704231 5089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:37:44.708057 5089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:37:44.708101 5089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:37:44.708150 5089 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:37:44.712623 5089 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:37:44.713541 5089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:37:44.714751 5089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:37:44.723441 5094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:44.723497 5095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:44.723505 5097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:44.723696 5089 server_base.cc:1034] running on GCE node
I20260619 14:37:44.724056 5089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:37:44.724642 5089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:37:44.725783 5089 hybrid_clock.cc:648] HybridClock initialized: now 1781879864725759 us; error 36 us; skew 500 ppm
I20260619 14:37:44.727380 5089 webserver.cc:460] Webserver started at http://127.0.98.1:33767/ using document root <none> and password file <none>
I20260619 14:37:44.727888 5089 fs_manager.cc:359] Metadata directory not provided
I20260619 14:37:44.727949 5089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:37:44.728130 5089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:37:44.730021 5089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/ts-0/data/instance:
uuid: "f2b9466dc329450cb9e2e8a693ed8497"
format_stamp: "Formatted at 2026-06-19 14:37:44 on dist-test-slave-32hj"
server_key: "bc04a4012c2339d8ee66534d349596f7"
server_key_iv: "2ac0ec19ceff094d8919753432e75e4f"
server_key_version: "encryptionkey@0"
I20260619 14:37:44.730515 5089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f2b9466dc329450cb9e2e8a693ed8497"
format_stamp: "Formatted at 2026-06-19 14:37:44 on dist-test-slave-32hj"
server_key: "bc04a4012c2339d8ee66534d349596f7"
server_key_iv: "2ac0ec19ceff094d8919753432e75e4f"
server_key_version: "encryptionkey@0"
I20260619 14:37:44.733949 5089 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:37:44.736218 5103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:44.737478 5089 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:37:44.737584 5089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/ts-0/wal
uuid: "f2b9466dc329450cb9e2e8a693ed8497"
format_stamp: "Formatted at 2026-06-19 14:37:44 on dist-test-slave-32hj"
server_key: "bc04a4012c2339d8ee66534d349596f7"
server_key_iv: "2ac0ec19ceff094d8919753432e75e4f"
server_key_version: "encryptionkey@0"
I20260619 14:37:44.737675 5089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:37:44.741902 5089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:37:44.742439 5089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:37:44.742579 5089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:37:44.743057 5089 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:37:44.743865 5089 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:37:44.743908 5089 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:44.743954 5089 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:37:44.743980 5089 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:44.751636 5089 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36223
I20260619 14:37:44.751680 5216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36223 every 8 connection(s)
I20260619 14:37:44.752424 5089 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:37:44.757710 5068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:44.754045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54347 (local address 127.0.98.62:36715)
0619 14:37:44.754198 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:44.754203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:44.755167 (+ 964us) server_negotiation.cc:407] Connection header received
0619 14:37:44.756341 (+ 1174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:44.756343 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:44.756377 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:44.756445 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:44.757388 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:44.757391 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:44.757393 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:37:44.757404 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:44.757423 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:37:44.757431 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:44.757497 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:44.757581 (+ 84us) server_negotiation.cc:299] Negotiation successful
0619 14:37:44.757613 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 14:37:44.758457 5219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:44.754427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36715 (local address 127.0.98.1:54347)
0619 14:37:44.755032 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:44.755068 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:44.756168 (+ 1100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:44.756578 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:44.756585 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:44.756665 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:44.757172 (+ 507us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:37:44.757174 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:44.757210 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:37:44.757270 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:44.757488 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:44.757493 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:44.757496 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 14:37:44.757606 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 14:37:44.757803 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":99,"threads_started":1}
I20260619 14:37:44.759454 5217 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36715
I20260619 14:37:44.759625 5217 heartbeater.cc:464] Registering TS with master...
I20260619 14:37:44.759955 5217 heartbeater.cc:511] Master 127.0.98.62:36715 requested a full tablet report, sending...
I20260619 14:37:44.761423 5029 ts_manager.cc:194] Registered new tserver with Master: f2b9466dc329450cb9e2e8a693ed8497 (127.0.98.1:36223)
I20260619 14:37:44.762540 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 5089
I20260619 14:37:44.762665 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:37:44.762898 392 external_mini_cluster.cc:1428] Setting key 962e8e2b060913f2c44c79671ebfbcdd
I20260619 14:37:44.764634 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36715
--builtin_ntp_servers=127.0.98.20:35815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 14:37:44.868678 5220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:37:44.868911 5220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:37:44.868947 5220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:37:44.872766 5220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:37:44.872804 5220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:37:44.872854 5220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:37:44.877233 5220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:37:44.878052 5220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:37:44.879205 5220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:37:44.888674 5226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:44.888680 5225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:44.888733 5228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:44.889132 5220 server_base.cc:1034] running on GCE node
I20260619 14:37:44.889493 5220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:37:44.890036 5220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:37:44.891173 5220 hybrid_clock.cc:648] HybridClock initialized: now 1781879864891142 us; error 47 us; skew 500 ppm
I20260619 14:37:44.892705 5220 webserver.cc:460] Webserver started at http://127.0.98.2:41861/ using document root <none> and password file <none>
I20260619 14:37:44.893251 5220 fs_manager.cc:359] Metadata directory not provided
I20260619 14:37:44.893306 5220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:37:44.893471 5220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:37:44.895313 5220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/ts-1/data/instance:
uuid: "758a7b9710a4463b8fb113f1900adea9"
format_stamp: "Formatted at 2026-06-19 14:37:44 on dist-test-slave-32hj"
server_key: "78c61841123da62408db2697d56aaa3b"
server_key_iv: "4c2665b25a1132583c85c6922fd16115"
server_key_version: "encryptionkey@0"
I20260619 14:37:44.895797 5220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/ts-1/wal/instance:
uuid: "758a7b9710a4463b8fb113f1900adea9"
format_stamp: "Formatted at 2026-06-19 14:37:44 on dist-test-slave-32hj"
server_key: "78c61841123da62408db2697d56aaa3b"
server_key_iv: "4c2665b25a1132583c85c6922fd16115"
server_key_version: "encryptionkey@0"
I20260619 14:37:44.899103 5220 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260619 14:37:44.901371 5234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:44.902495 5220 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:37:44.902588 5220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/ts-1/wal
uuid: "758a7b9710a4463b8fb113f1900adea9"
format_stamp: "Formatted at 2026-06-19 14:37:44 on dist-test-slave-32hj"
server_key: "78c61841123da62408db2697d56aaa3b"
server_key_iv: "4c2665b25a1132583c85c6922fd16115"
server_key_version: "encryptionkey@0"
I20260619 14:37:44.902671 5220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:37:44.906704 5220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:37:44.907217 5220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:37:44.907356 5220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:37:44.907831 5220 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:37:44.908617 5220 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:37:44.908660 5220 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:44.908704 5220 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:37:44.908771 5220 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:44.916406 5220 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35327
I20260619 14:37:44.916432 5347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35327 every 8 connection(s)
I20260619 14:37:44.917212 5220 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:37:44.919881 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 5220
I20260619 14:37:44.919986 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:37:44.920203 392 external_mini_cluster.cc:1428] Setting key 52ec326b38178c0e22f10cbdff408011
I20260619 14:37:44.921983 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36715
--builtin_ntp_servers=127.0.98.20:35815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 14:37:44.922850 5068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:44.918781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33159 (local address 127.0.98.62:36715)
0619 14:37:44.918911 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:44.918915 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:44.919900 (+ 985us) server_negotiation.cc:407] Connection header received
0619 14:37:44.921333 (+ 1433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:44.921336 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:44.921370 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:44.921456 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:44.922473 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:44.922477 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:44.922479 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:37:44.922491 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:44.922511 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:37:44.922519 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:44.922595 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:44.922696 (+ 101us) server_negotiation.cc:299] Negotiation successful
0619 14:37:44.922732 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 14:37:44.923585 5350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:44.919141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36715 (local address 127.0.98.2:33159)
0619 14:37:44.919754 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:44.919795 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:44.921127 (+ 1332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:44.921603 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:44.921610 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:44.921691 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:44.922226 (+ 535us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:37:44.922227 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:44.922263 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:37:44.922344 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:44.922593 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:44.922598 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:44.922601 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 14:37:44.922709 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 14:37:44.922881 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":101,"threads_started":1}
I20260619 14:37:44.924615 5348 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36715
I20260619 14:37:44.924844 5348 heartbeater.cc:464] Registering TS with master...
I20260619 14:37:44.925195 5348 heartbeater.cc:511] Master 127.0.98.62:36715 requested a full tablet report, sending...
I20260619 14:37:44.926127 5029 ts_manager.cc:194] Registered new tserver with Master: 758a7b9710a4463b8fb113f1900adea9 (127.0.98.2:35327)
W20260619 14:37:45.028412 5351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:37:45.028869 5351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:37:45.028955 5351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:37:45.032927 5351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:37:45.032987 5351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:37:45.033056 5351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:37:45.037765 5351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:37:45.038761 5351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:37:45.040073 5351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:37:45.049037 5351 server_base.cc:1034] running on GCE node
W20260619 14:37:45.049005 5356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:45.048975 5357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:45.048998 5359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:45.049620 5351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:37:45.050200 5351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:37:45.051363 5351 hybrid_clock.cc:648] HybridClock initialized: now 1781879865051344 us; error 32 us; skew 500 ppm
I20260619 14:37:45.053062 5351 webserver.cc:460] Webserver started at http://127.0.98.3:46021/ using document root <none> and password file <none>
I20260619 14:37:45.053574 5351 fs_manager.cc:359] Metadata directory not provided
I20260619 14:37:45.053630 5351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:37:45.053815 5351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:37:45.055719 5351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/ts-2/data/instance:
uuid: "eca6abc7e70e4d5595a0b8d8b6393ef8"
format_stamp: "Formatted at 2026-06-19 14:37:45 on dist-test-slave-32hj"
server_key: "f96c49bcf0e9798ed3e6f7a0cecae09e"
server_key_iv: "260ca9414532f7bb00558a48ac534c16"
server_key_version: "encryptionkey@0"
I20260619 14:37:45.056222 5351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/ts-2/wal/instance:
uuid: "eca6abc7e70e4d5595a0b8d8b6393ef8"
format_stamp: "Formatted at 2026-06-19 14:37:45 on dist-test-slave-32hj"
server_key: "f96c49bcf0e9798ed3e6f7a0cecae09e"
server_key_iv: "260ca9414532f7bb00558a48ac534c16"
server_key_version: "encryptionkey@0"
I20260619 14:37:45.059731 5351 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 14:37:45.062256 5365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:45.063408 5351 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:37:45.063519 5351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/ts-2/wal
uuid: "eca6abc7e70e4d5595a0b8d8b6393ef8"
format_stamp: "Formatted at 2026-06-19 14:37:45 on dist-test-slave-32hj"
server_key: "f96c49bcf0e9798ed3e6f7a0cecae09e"
server_key_iv: "260ca9414532f7bb00558a48ac534c16"
server_key_version: "encryptionkey@0"
I20260619 14:37:45.063616 5351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:37:45.067860 5351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:37:45.068393 5351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:37:45.068542 5351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:37:45.069056 5351 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:37:45.069877 5351 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:37:45.069921 5351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:45.069960 5351 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:37:45.069989 5351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:45.077725 5351 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41121
I20260619 14:37:45.077771 5478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41121 every 8 connection(s)
I20260619 14:37:45.078568 5351 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:37:45.084146 5068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:45.080196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49323 (local address 127.0.98.62:36715)
0619 14:37:45.080318 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:45.080322 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:45.081355 (+ 1033us) server_negotiation.cc:407] Connection header received
0619 14:37:45.082583 (+ 1228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:45.082586 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:45.082628 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:45.082704 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:45.083704 (+ 1000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:45.083707 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:45.083710 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:37:45.083723 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:45.083743 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:37:45.083751 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:45.083826 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:45.083970 (+ 144us) server_negotiation.cc:299] Negotiation successful
0619 14:37:45.084015 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 14:37:45.085000 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:45.080572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36715 (local address 127.0.98.3:49323)
0619 14:37:45.081215 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:45.081254 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:45.082379 (+ 1125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:45.082880 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:45.082890 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:45.082990 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:45.083464 (+ 474us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:37:45.083465 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:45.083503 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:37:45.083567 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:45.083817 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:45.083824 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:45.083831 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 14:37:45.084011 (+ 180us) client_negotiation.cc:240] Negotiation successful
0619 14:37:45.084247 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":135,"threads_started":1}
I20260619 14:37:45.085989 5479 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36715
I20260619 14:37:45.086194 5479 heartbeater.cc:464] Registering TS with master...
I20260619 14:37:45.086560 5479 heartbeater.cc:511] Master 127.0.98.62:36715 requested a full tablet report, sending...
I20260619 14:37:45.087515 5029 ts_manager.cc:194] Registered new tserver with Master: eca6abc7e70e4d5595a0b8d8b6393ef8 (127.0.98.3:41121)
I20260619 14:37:45.087682 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 5351
I20260619 14:37:45.087787 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:37:45.088099 392 external_mini_cluster.cc:1428] Setting key d3466396dac353a4f9ccdd8ae4e0cab4
I20260619 14:37:45.089879 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:37:45.093024 5068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:45.091598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53306 (local address 127.0.98.62:36715)
0619 14:37:45.091720 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:45.091724 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:45.091884 (+ 160us) server_negotiation.cc:407] Connection header received
0619 14:37:45.092001 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:45.092003 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:45.092034 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:45.092095 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:45.092646 (+ 551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:45.092650 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:45.092653 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:37:45.092663 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:45.092681 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:37:45.092688 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:45.092798 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:45.092858 (+ 60us) server_negotiation.cc:299] Negotiation successful
0619 14:37:45.092887 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 14:37:45.095575 5029 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53306:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:37:45.098464 5029 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:37:45.109038 5491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:45.106088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41121 (local address 127.0.0.1:48870)
0619 14:37:45.106615 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:45.106626 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:45.106686 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:45.107574 (+ 888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:45.107577 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:45.107593 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:45.107955 (+ 362us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:37:45.107957 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:45.107968 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:37:45.107993 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:45.108792 (+ 799us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:45.108795 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:45.108798 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 14:37:45.108850 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 14:37:45.108884 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":74,"threads_started":1}
I20260619 14:37:45.109040 5490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:45.105980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35327 (local address 127.0.0.1:50110)
0619 14:37:45.106290 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:45.106340 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:45.106435 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:45.107253 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:45.107260 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:45.107338 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:45.107825 (+ 487us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:37:45.107827 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:45.107847 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:37:45.107914 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:45.108662 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:45.108670 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:45.108676 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 14:37:45.108794 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 14:37:45.108858 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":78,"threads_started":1}
I20260619 14:37:45.109329 5494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:45.107137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36223 (local address 127.0.0.1:48512)
0619 14:37:45.107384 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:45.107396 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:45.107481 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:45.108296 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:45.108299 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:45.108313 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:45.108757 (+ 444us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:37:45.108758 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:45.108768 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:37:45.108793 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:45.109127 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:45.109130 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:45.109133 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 14:37:45.109201 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 14:37:45.109230 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":87,"threads_started":1}
I20260619 14:37:45.109632 5492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:45.106094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50110 (local address 127.0.98.2:35327)
0619 14:37:45.106535 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:45.106547 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:45.106572 (+ 25us) server_negotiation.cc:407] Connection header received
0619 14:37:45.106853 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:45.106866 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:45.107093 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:45.107610 (+ 517us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:45.108474 (+ 864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:45.108480 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:45.108492 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:37:45.108502 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:45.108538 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:37:45.108563 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:45.108981 (+ 418us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:45.109025 (+ 44us) server_negotiation.cc:299] Negotiation successful
0619 14:37:45.109470 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":120,"threads_started":1}
I20260619 14:37:45.109647 5493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:45.106711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48870 (local address 127.0.98.3:41121)
0619 14:37:45.107143 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:45.107154 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:45.107178 (+ 24us) server_negotiation.cc:407] Connection header received
0619 14:37:45.107230 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:45.107236 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:45.107416 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:45.107789 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:45.108574 (+ 785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:45.108579 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:45.108590 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:37:45.108599 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:45.108633 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:37:45.108654 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:45.109363 (+ 709us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:45.109410 (+ 47us) server_negotiation.cc:299] Negotiation successful
0619 14:37:45.109490 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":143,"threads_started":1}
I20260619 14:37:45.109709 5495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:45.107507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48512 (local address 127.0.98.1:36223)
0619 14:37:45.107899 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:45.107909 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:45.107944 (+ 35us) server_negotiation.cc:407] Connection header received
0619 14:37:45.107993 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:45.107998 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:45.108165 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:45.108293 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:45.108900 (+ 607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:45.108905 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:45.108914 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:37:45.108926 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:45.108971 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:37:45.108995 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:45.109087 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:45.109509 (+ 422us) server_negotiation.cc:299] Negotiation successful
0619 14:37:45.109567 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":103,"threads_started":1}
I20260619 14:37:45.112169 5413 tablet_service.cc:1467] Processing CreateTablet for tablet 9c958bfe7bd049d1afc29bfe8bf66726 (DEFAULT_TABLE table=test-table [id=cbbd3d23d19444fc9e14c06026314e1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:37:45.112167 5282 tablet_service.cc:1467] Processing CreateTablet for tablet 9c958bfe7bd049d1afc29bfe8bf66726 (DEFAULT_TABLE table=test-table [id=cbbd3d23d19444fc9e14c06026314e1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:37:45.113193 5413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c958bfe7bd049d1afc29bfe8bf66726. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:45.113191 5282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c958bfe7bd049d1afc29bfe8bf66726. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:45.114609 5151 tablet_service.cc:1467] Processing CreateTablet for tablet 9c958bfe7bd049d1afc29bfe8bf66726 (DEFAULT_TABLE table=test-table [id=cbbd3d23d19444fc9e14c06026314e1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:37:45.115572 5151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c958bfe7bd049d1afc29bfe8bf66726. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:45.118988 5496 tablet_bootstrap.cc:492] T 9c958bfe7bd049d1afc29bfe8bf66726 P eca6abc7e70e4d5595a0b8d8b6393ef8: Bootstrap starting.
I20260619 14:37:45.120402 5497 tablet_bootstrap.cc:492] T 9c958bfe7bd049d1afc29bfe8bf66726 P 758a7b9710a4463b8fb113f1900adea9: Bootstrap starting.
I20260619 14:37:45.121773 5496 tablet_bootstrap.cc:654] T 9c958bfe7bd049d1afc29bfe8bf66726 P eca6abc7e70e4d5595a0b8d8b6393ef8: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:45.122272 5498 tablet_bootstrap.cc:492] T 9c958bfe7bd049d1afc29bfe8bf66726 P f2b9466dc329450cb9e2e8a693ed8497: Bootstrap starting.
I20260619 14:37:45.122531 5497 tablet_bootstrap.cc:654] T 9c958bfe7bd049d1afc29bfe8bf66726 P 758a7b9710a4463b8fb113f1900adea9: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:45.122674 5496 log.cc:824] T 9c958bfe7bd049d1afc29bfe8bf66726 P eca6abc7e70e4d5595a0b8d8b6393ef8: Log is configured to *not* fsync() on all Append() calls
I20260619 14:37:45.123299 5497 log.cc:824] T 9c958bfe7bd049d1afc29bfe8bf66726 P 758a7b9710a4463b8fb113f1900adea9: Log is configured to *not* fsync() on all Append() calls
I20260619 14:37:45.124377 5498 tablet_bootstrap.cc:654] T 9c958bfe7bd049d1afc29bfe8bf66726 P f2b9466dc329450cb9e2e8a693ed8497: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:45.124760 5496 tablet_bootstrap.cc:492] T 9c958bfe7bd049d1afc29bfe8bf66726 P eca6abc7e70e4d5595a0b8d8b6393ef8: No bootstrap required, opened a new log
I20260619 14:37:45.124955 5496 ts_tablet_manager.cc:1397] T 9c958bfe7bd049d1afc29bfe8bf66726 P eca6abc7e70e4d5595a0b8d8b6393ef8: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 14:37:45.125109 5498 log.cc:824] T 9c958bfe7bd049d1afc29bfe8bf66726 P f2b9466dc329450cb9e2e8a693ed8497: Log is configured to *not* fsync() on all Append() calls
I20260619 14:37:45.125202 5497 tablet_bootstrap.cc:492] T 9c958bfe7bd049d1afc29bfe8bf66726 P 758a7b9710a4463b8fb113f1900adea9: No bootstrap required, opened a new log
I20260619 14:37:45.125371 5497 ts_tablet_manager.cc:1397] T 9c958bfe7bd049d1afc29bfe8bf66726 P 758a7b9710a4463b8fb113f1900adea9: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:37:45.126714 5498 tablet_bootstrap.cc:492] T 9c958bfe7bd049d1afc29bfe8bf66726 P f2b9466dc329450cb9e2e8a693ed8497: No bootstrap required, opened a new log
I20260619 14:37:45.127066 5498 ts_tablet_manager.cc:1397] T 9c958bfe7bd049d1afc29bfe8bf66726 P f2b9466dc329450cb9e2e8a693ed8497: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:37:45.129537 5497 raft_consensus.cc:348] T 9c958bfe7bd049d1afc29bfe8bf66726 P 758a7b9710a4463b8fb113f1900adea9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758a7b9710a4463b8fb113f1900adea9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35327 } } peers { permanent_uuid: "eca6abc7e70e4d5595a0b8d8b6393ef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41121 } } peers { permanent_uuid: "f2b9466dc329450cb9e2e8a693ed8497" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } }
I20260619 14:37:45.129537 5496 raft_consensus.cc:348] T 9c958bfe7bd049d1afc29bfe8bf66726 P eca6abc7e70e4d5595a0b8d8b6393ef8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758a7b9710a4463b8fb113f1900adea9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35327 } } peers { permanent_uuid: "eca6abc7e70e4d5595a0b8d8b6393ef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41121 } } peers { permanent_uuid: "f2b9466dc329450cb9e2e8a693ed8497" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } }
I20260619 14:37:45.129820 5497 raft_consensus.cc:374] T 9c958bfe7bd049d1afc29bfe8bf66726 P 758a7b9710a4463b8fb113f1900adea9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:45.129820 5496 raft_consensus.cc:374] T 9c958bfe7bd049d1afc29bfe8bf66726 P eca6abc7e70e4d5595a0b8d8b6393ef8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:45.129875 5496 raft_consensus.cc:729] T 9c958bfe7bd049d1afc29bfe8bf66726 P eca6abc7e70e4d5595a0b8d8b6393ef8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eca6abc7e70e4d5595a0b8d8b6393ef8, State: Initialized, Role: FOLLOWER
I20260619 14:37:45.129882 5497 raft_consensus.cc:729] T 9c958bfe7bd049d1afc29bfe8bf66726 P 758a7b9710a4463b8fb113f1900adea9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 758a7b9710a4463b8fb113f1900adea9, State: Initialized, Role: FOLLOWER
I20260619 14:37:45.130347 5497 consensus_queue.cc:260] T 9c958bfe7bd049d1afc29bfe8bf66726 P 758a7b9710a4463b8fb113f1900adea9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758a7b9710a4463b8fb113f1900adea9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35327 } } peers { permanent_uuid: "eca6abc7e70e4d5595a0b8d8b6393ef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41121 } } peers { permanent_uuid: "f2b9466dc329450cb9e2e8a693ed8497" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } }
I20260619 14:37:45.130347 5496 consensus_queue.cc:260] T 9c958bfe7bd049d1afc29bfe8bf66726 P eca6abc7e70e4d5595a0b8d8b6393ef8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758a7b9710a4463b8fb113f1900adea9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35327 } } peers { permanent_uuid: "eca6abc7e70e4d5595a0b8d8b6393ef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41121 } } peers { permanent_uuid: "f2b9466dc329450cb9e2e8a693ed8497" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } }
I20260619 14:37:45.131037 5348 heartbeater.cc:503] Master 127.0.98.62:36715 was elected leader, sending a full tablet report...
I20260619 14:37:45.131301 5479 heartbeater.cc:503] Master 127.0.98.62:36715 was elected leader, sending a full tablet report...
I20260619 14:37:45.131395 5497 ts_tablet_manager.cc:1428] T 9c958bfe7bd049d1afc29bfe8bf66726 P 758a7b9710a4463b8fb113f1900adea9: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 14:37:45.131902 5498 raft_consensus.cc:348] T 9c958bfe7bd049d1afc29bfe8bf66726 P f2b9466dc329450cb9e2e8a693ed8497 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758a7b9710a4463b8fb113f1900adea9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35327 } } peers { permanent_uuid: "eca6abc7e70e4d5595a0b8d8b6393ef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41121 } } peers { permanent_uuid: "f2b9466dc329450cb9e2e8a693ed8497" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } }
I20260619 14:37:45.132189 5498 raft_consensus.cc:374] T 9c958bfe7bd049d1afc29bfe8bf66726 P f2b9466dc329450cb9e2e8a693ed8497 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:45.132251 5498 raft_consensus.cc:729] T 9c958bfe7bd049d1afc29bfe8bf66726 P f2b9466dc329450cb9e2e8a693ed8497 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2b9466dc329450cb9e2e8a693ed8497, State: Initialized, Role: FOLLOWER
I20260619 14:37:45.132371 5496 ts_tablet_manager.cc:1428] T 9c958bfe7bd049d1afc29bfe8bf66726 P eca6abc7e70e4d5595a0b8d8b6393ef8: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260619 14:37:45.132764 5498 consensus_queue.cc:260] T 9c958bfe7bd049d1afc29bfe8bf66726 P f2b9466dc329450cb9e2e8a693ed8497 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758a7b9710a4463b8fb113f1900adea9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35327 } } peers { permanent_uuid: "eca6abc7e70e4d5595a0b8d8b6393ef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41121 } } peers { permanent_uuid: "f2b9466dc329450cb9e2e8a693ed8497" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } }
I20260619 14:37:45.133788 5498 ts_tablet_manager.cc:1428] T 9c958bfe7bd049d1afc29bfe8bf66726 P f2b9466dc329450cb9e2e8a693ed8497: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 14:37:45.134115 5217 heartbeater.cc:503] Master 127.0.98.62:36715 was elected leader, sending a full tablet report...
I20260619 14:37:45.159624 5502 raft_consensus.cc:482] T 9c958bfe7bd049d1afc29bfe8bf66726 P 758a7b9710a4463b8fb113f1900adea9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:37:45.159888 5502 raft_consensus.cc:504] T 9c958bfe7bd049d1afc29bfe8bf66726 P 758a7b9710a4463b8fb113f1900adea9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758a7b9710a4463b8fb113f1900adea9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35327 } } peers { permanent_uuid: "eca6abc7e70e4d5595a0b8d8b6393ef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41121 } } peers { permanent_uuid: "f2b9466dc329450cb9e2e8a693ed8497" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } }
I20260619 14:37:45.161105 5502 leader_election.cc:288] T 9c958bfe7bd049d1afc29bfe8bf66726 P 758a7b9710a4463b8fb113f1900adea9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eca6abc7e70e4d5595a0b8d8b6393ef8 (127.0.98.3:41121), f2b9466dc329450cb9e2e8a693ed8497 (127.0.98.1:36223)
I20260619 14:37:45.166199 5495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:45.161757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40337 (local address 127.0.98.1:36223)
0619 14:37:45.161944 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:45.161948 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:45.162110 (+ 162us) server_negotiation.cc:407] Connection header received
0619 14:37:45.162185 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:45.162187 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:45.162224 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:45.162308 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:45.165841 (+ 3533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:45.165845 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:45.165848 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:37:45.165860 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:45.165878 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:37:45.165885 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:45.165956 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:45.166030 (+ 74us) server_negotiation.cc:299] Negotiation successful
0619 14:37:45.166072 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260619 14:37:45.166199 5350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:45.161342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41121 (local address 127.0.98.2:44341)
0619 14:37:45.161458 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:45.161474 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:45.161554 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:45.161856 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:45.161860 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:45.161878 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:45.165686 (+ 3808us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:37:45.165688 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:45.165700 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:37:45.165731 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:45.165993 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:45.165996 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:45.165999 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 14:37:45.166072 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 14:37:45.166103 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260619 14:37:45.166224 5505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:45.161601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36223 (local address 127.0.98.2:40337)
0619 14:37:45.162008 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:45.162020 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:45.162113 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:45.162335 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:45.162338 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:45.162353 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:45.165696 (+ 3343us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:37:45.165698 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:45.165707 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:37:45.165732 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:45.165956 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:45.165959 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:45.165962 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 14:37:45.166032 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 14:37:45.166068 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":145,"threads_started":1}
I20260619 14:37:45.166219 5493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:45.161432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44341 (local address 127.0.98.3:41121)
0619 14:37:45.161552 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:45.161558 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:45.161577 (+ 19us) server_negotiation.cc:407] Connection header received
0619 14:37:45.161631 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:45.161634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:45.161675 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:45.161790 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:45.165865 (+ 4075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:45.165868 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:45.165870 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:37:45.165882 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:45.165900 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:37:45.165908 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:45.166001 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:45.166079 (+ 78us) server_negotiation.cc:299] Negotiation successful
0619 14:37:45.166115 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 14:37:45.166828 5171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c958bfe7bd049d1afc29bfe8bf66726" candidate_uuid: "758a7b9710a4463b8fb113f1900adea9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2b9466dc329450cb9e2e8a693ed8497" is_pre_election: true
I20260619 14:37:45.166846 5433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c958bfe7bd049d1afc29bfe8bf66726" candidate_uuid: "758a7b9710a4463b8fb113f1900adea9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eca6abc7e70e4d5595a0b8d8b6393ef8" is_pre_election: true
I20260619 14:37:45.167097 5433 raft_consensus.cc:2454] T 9c958bfe7bd049d1afc29bfe8bf66726 P eca6abc7e70e4d5595a0b8d8b6393ef8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 758a7b9710a4463b8fb113f1900adea9 in term 0.
I20260619 14:37:45.167110 5171 raft_consensus.cc:2454] T 9c958bfe7bd049d1afc29bfe8bf66726 P f2b9466dc329450cb9e2e8a693ed8497 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 758a7b9710a4463b8fb113f1900adea9 in term 0.
I20260619 14:37:45.167516 5238 leader_election.cc:302] T 9c958bfe7bd049d1afc29bfe8bf66726 P 758a7b9710a4463b8fb113f1900adea9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 758a7b9710a4463b8fb113f1900adea9, eca6abc7e70e4d5595a0b8d8b6393ef8; no voters:
I20260619 14:37:45.167795 5502 raft_consensus.cc:2781] T 9c958bfe7bd049d1afc29bfe8bf66726 P 758a7b9710a4463b8fb113f1900adea9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:37:45.167929 5502 raft_consensus.cc:482] T 9c958bfe7bd049d1afc29bfe8bf66726 P 758a7b9710a4463b8fb113f1900adea9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:37:45.167984 5502 raft_consensus.cc:3037] T 9c958bfe7bd049d1afc29bfe8bf66726 P 758a7b9710a4463b8fb113f1900adea9 [term 0 FOLLOWER]: Advancing to term 1
W20260619 14:37:45.168018 5349 tablet.cc:2369] T 9c958bfe7bd049d1afc29bfe8bf66726 P 758a7b9710a4463b8fb113f1900adea9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:37:45.169098 5502 raft_consensus.cc:504] T 9c958bfe7bd049d1afc29bfe8bf66726 P 758a7b9710a4463b8fb113f1900adea9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758a7b9710a4463b8fb113f1900adea9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35327 } } peers { permanent_uuid: "eca6abc7e70e4d5595a0b8d8b6393ef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41121 } } peers { permanent_uuid: "f2b9466dc329450cb9e2e8a693ed8497" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } }
I20260619 14:37:45.169375 5502 leader_election.cc:288] T 9c958bfe7bd049d1afc29bfe8bf66726 P 758a7b9710a4463b8fb113f1900adea9 [CANDIDATE]: Term 1 election: Requested vote from peers eca6abc7e70e4d5595a0b8d8b6393ef8 (127.0.98.3:41121), f2b9466dc329450cb9e2e8a693ed8497 (127.0.98.1:36223)
I20260619 14:37:45.169667 5433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c958bfe7bd049d1afc29bfe8bf66726" candidate_uuid: "758a7b9710a4463b8fb113f1900adea9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eca6abc7e70e4d5595a0b8d8b6393ef8"
I20260619 14:37:45.169790 5433 raft_consensus.cc:3037] T 9c958bfe7bd049d1afc29bfe8bf66726 P eca6abc7e70e4d5595a0b8d8b6393ef8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:45.169802 5171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c958bfe7bd049d1afc29bfe8bf66726" candidate_uuid: "758a7b9710a4463b8fb113f1900adea9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2b9466dc329450cb9e2e8a693ed8497"
I20260619 14:37:45.169909 5171 raft_consensus.cc:3037] T 9c958bfe7bd049d1afc29bfe8bf66726 P f2b9466dc329450cb9e2e8a693ed8497 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:45.171036 5433 raft_consensus.cc:2454] T 9c958bfe7bd049d1afc29bfe8bf66726 P eca6abc7e70e4d5595a0b8d8b6393ef8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 758a7b9710a4463b8fb113f1900adea9 in term 1.
I20260619 14:37:45.171036 5171 raft_consensus.cc:2454] T 9c958bfe7bd049d1afc29bfe8bf66726 P f2b9466dc329450cb9e2e8a693ed8497 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 758a7b9710a4463b8fb113f1900adea9 in term 1.
I20260619 14:37:45.171285 5238 leader_election.cc:302] T 9c958bfe7bd049d1afc29bfe8bf66726 P 758a7b9710a4463b8fb113f1900adea9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 758a7b9710a4463b8fb113f1900adea9, f2b9466dc329450cb9e2e8a693ed8497; no voters:
I20260619 14:37:45.171455 5502 raft_consensus.cc:2781] T 9c958bfe7bd049d1afc29bfe8bf66726 P 758a7b9710a4463b8fb113f1900adea9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:37:45.171657 5502 raft_consensus.cc:686] T 9c958bfe7bd049d1afc29bfe8bf66726 P 758a7b9710a4463b8fb113f1900adea9 [term 1 LEADER]: Becoming Leader. State: Replica: 758a7b9710a4463b8fb113f1900adea9, State: Running, Role: LEADER
I20260619 14:37:45.171950 5502 consensus_queue.cc:237] T 9c958bfe7bd049d1afc29bfe8bf66726 P 758a7b9710a4463b8fb113f1900adea9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758a7b9710a4463b8fb113f1900adea9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35327 } } peers { permanent_uuid: "eca6abc7e70e4d5595a0b8d8b6393ef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41121 } } peers { permanent_uuid: "f2b9466dc329450cb9e2e8a693ed8497" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } }
I20260619 14:37:45.174909 5027 catalog_manager.cc:5495] T 9c958bfe7bd049d1afc29bfe8bf66726 P 758a7b9710a4463b8fb113f1900adea9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 758a7b9710a4463b8fb113f1900adea9 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "758a7b9710a4463b8fb113f1900adea9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "758a7b9710a4463b8fb113f1900adea9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35327 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eca6abc7e70e4d5595a0b8d8b6393ef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2b9466dc329450cb9e2e8a693ed8497" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 } health_report { overall_health: UNKNOWN } } }
I20260619 14:37:45.186257 5492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:45.184892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50120 (local address 127.0.98.2:35327)
0619 14:37:45.185001 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:45.185006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:45.185039 (+ 33us) server_negotiation.cc:407] Connection header received
0619 14:37:45.185203 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:45.185206 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:45.185262 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:45.185329 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:45.185905 (+ 576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:45.185909 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:45.185912 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:37:45.185923 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:45.185943 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:37:45.185952 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:45.186027 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:45.186101 (+ 74us) server_negotiation.cc:299] Negotiation successful
0619 14:37:45.186136 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 14:37:45.191601 5171 raft_consensus.cc:1261] T 9c958bfe7bd049d1afc29bfe8bf66726 P f2b9466dc329450cb9e2e8a693ed8497 [term 1 FOLLOWER]: Refusing update from remote peer 758a7b9710a4463b8fb113f1900adea9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:37:45.191601 5433 raft_consensus.cc:1261] T 9c958bfe7bd049d1afc29bfe8bf66726 P eca6abc7e70e4d5595a0b8d8b6393ef8 [term 1 FOLLOWER]: Refusing update from remote peer 758a7b9710a4463b8fb113f1900adea9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:37:45.192247 5506 consensus_queue.cc:1035] T 9c958bfe7bd049d1afc29bfe8bf66726 P 758a7b9710a4463b8fb113f1900adea9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eca6abc7e70e4d5595a0b8d8b6393ef8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:37:45.192399 5502 consensus_queue.cc:1035] T 9c958bfe7bd049d1afc29bfe8bf66726 P 758a7b9710a4463b8fb113f1900adea9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2b9466dc329450cb9e2e8a693ed8497" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:37:45.201000 5510 mvcc.cc:205] Tried to move back new op lower bound from 7298579927818080256 to 7298579927746691072. Current Snapshot: MvccSnapshot[applied={T|T < 7298579927818080256}]
I20260619 14:37:45.208102 5028 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:37:45.208242 5028 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:37:45.210368 5028 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3b3447ac5b3343878b45d540f9576301: Sending DeleteTablet for 3 replicas of tablet 9c958bfe7bd049d1afc29bfe8bf66726
I20260619 14:37:45.211022 5282 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c958bfe7bd049d1afc29bfe8bf66726 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:37:45 UTC) from {username='slave'} at 127.0.0.1:50110
I20260619 14:37:45.211161 5151 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c958bfe7bd049d1afc29bfe8bf66726 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:37:45 UTC) from {username='slave'} at 127.0.0.1:48512
I20260619 14:37:45.211512 5519 tablet_replica.cc:330] T 9c958bfe7bd049d1afc29bfe8bf66726 P 758a7b9710a4463b8fb113f1900adea9: stopping tablet replica
I20260619 14:37:45.211495 5413 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c958bfe7bd049d1afc29bfe8bf66726 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:37:45 UTC) from {username='slave'} at 127.0.0.1:48870
I20260619 14:37:45.211687 5520 tablet_replica.cc:330] T 9c958bfe7bd049d1afc29bfe8bf66726 P f2b9466dc329450cb9e2e8a693ed8497: stopping tablet replica
I20260619 14:37:45.211795 5519 raft_consensus.cc:2229] T 9c958bfe7bd049d1afc29bfe8bf66726 P 758a7b9710a4463b8fb113f1900adea9 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:37:45.212002 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 5089
I20260619 14:37:45.212169 5519 raft_consensus.cc:2258] T 9c958bfe7bd049d1afc29bfe8bf66726 P 758a7b9710a4463b8fb113f1900adea9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:37:45.213907 5521 tablet_replica.cc:330] T 9c958bfe7bd049d1afc29bfe8bf66726 P eca6abc7e70e4d5595a0b8d8b6393ef8: stopping tablet replica
I20260619 14:37:45.214010 5519 ts_tablet_manager.cc:1905] T 9c958bfe7bd049d1afc29bfe8bf66726 P 758a7b9710a4463b8fb113f1900adea9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:37:45.214196 5521 raft_consensus.cc:2229] T 9c958bfe7bd049d1afc29bfe8bf66726 P eca6abc7e70e4d5595a0b8d8b6393ef8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:37:45.214480 5521 raft_consensus.cc:2258] T 9c958bfe7bd049d1afc29bfe8bf66726 P eca6abc7e70e4d5595a0b8d8b6393ef8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:37:45.215767 5521 ts_tablet_manager.cc:1905] T 9c958bfe7bd049d1afc29bfe8bf66726 P eca6abc7e70e4d5595a0b8d8b6393ef8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:37:45.216743 5519 ts_tablet_manager.cc:1918] T 9c958bfe7bd049d1afc29bfe8bf66726 P 758a7b9710a4463b8fb113f1900adea9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:37:45.216838 5519 log.cc:1196] T 9c958bfe7bd049d1afc29bfe8bf66726 P 758a7b9710a4463b8fb113f1900adea9: Deleting WAL directory at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/ts-1/wal/wals/9c958bfe7bd049d1afc29bfe8bf66726
I20260619 14:37:45.217150 5519 ts_tablet_manager.cc:1939] T 9c958bfe7bd049d1afc29bfe8bf66726 P 758a7b9710a4463b8fb113f1900adea9: Deleting consensus metadata
I20260619 14:37:45.218207 5521 ts_tablet_manager.cc:1918] T 9c958bfe7bd049d1afc29bfe8bf66726 P eca6abc7e70e4d5595a0b8d8b6393ef8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:37:45.218298 5521 log.cc:1196] T 9c958bfe7bd049d1afc29bfe8bf66726 P eca6abc7e70e4d5595a0b8d8b6393ef8: Deleting WAL directory at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781879837648036-392-0/minicluster-data/ts-2/wal/wals/9c958bfe7bd049d1afc29bfe8bf66726
I20260619 14:37:45.218330 5013 catalog_manager.cc:4841] TS 758a7b9710a4463b8fb113f1900adea9 (127.0.98.2:35327): tablet 9c958bfe7bd049d1afc29bfe8bf66726 (table test-table [id=cbbd3d23d19444fc9e14c06026314e1f]) successfully deleted
I20260619 14:37:45.218567 5521 ts_tablet_manager.cc:1939] T 9c958bfe7bd049d1afc29bfe8bf66726 P eca6abc7e70e4d5595a0b8d8b6393ef8: Deleting consensus metadata
I20260619 14:37:45.219508 5016 catalog_manager.cc:4841] TS eca6abc7e70e4d5595a0b8d8b6393ef8 (127.0.98.3:41121): tablet 9c958bfe7bd049d1afc29bfe8bf66726 (table test-table [id=cbbd3d23d19444fc9e14c06026314e1f]) successfully deleted
W20260619 14:37:45.219744 5016 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36223 (error 108)
I20260619 14:37:45.219825 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 5220
I20260619 14:37:45.221128 5494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:45.220794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36223 (local address 127.0.0.1:48516)
0619 14:37:45.220979 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:45.221045 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
W20260619 14:37:45.221287 5016 catalog_manager.cc:4584] TS f2b9466dc329450cb9e2e8a693ed8497 (127.0.98.1:36223): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9c958bfe7bd049d1afc29bfe8bf66726: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36223: connect: Connection refused (error 111)
I20260619 14:37:45.224620 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 5351
I20260619 14:37:45.228929 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 4998
2026-06-19T14:37:45Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (864 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260619 14:37:45.248512 392 mini_oidc.cc:152] Starting JWKS server
I20260619 14:37:45.250109 392 webserver.cc:460] Webserver started at https://127.0.0.1:39513/ using document root <none> and password file <none>
I20260619 14:37:45.250248 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 14:37:45.250465 392 webserver.cc:460] Webserver started at http://127.0.0.1:36361/ using document root <none> and password file <none>
2026-06-19T14:37:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:37:45Z Disabled control of system clock
I20260619 14:37:45.258788 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781879837648036-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781879837648036-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781879837648036-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41249
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41249
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39513/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781879837648036-392-0/testchainca.cert with env {}
W20260619 14:37:45.363560 5528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:37:45.363806 5528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:37:45.363844 5528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:37:45.367748 5528 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:37:45.367798 5528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:37:45.367812 5528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:37:45.367826 5528 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:37:45.367841 5528 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 14:37:45.367861 5528 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 14:37:45.367872 5528 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39513/jwks.json
I20260619 14:37:45.373088 5528 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781879837648036-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781879837648036-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41249
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39513/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781879837648036-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781879837648036-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:37:45.374040 5528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:37:45.374954 5528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:37:45.383380 5533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:45.383390 5536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:45.383350 5534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:45.383421 5528 server_base.cc:1034] running on GCE node
I20260619 14:37:45.384120 5528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:37:45.385187 5528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:37:45.386364 5528 hybrid_clock.cc:648] HybridClock initialized: now 1781879865386335 us; error 41 us; skew 500 ppm
I20260619 14:37:45.388088 5528 webserver.cc:460] Webserver started at http://127.0.98.62:45399/ using document root <none> and password file <none>
I20260619 14:37:45.388612 5528 fs_manager.cc:359] Metadata directory not provided
I20260619 14:37:45.388674 5528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:37:45.388891 5528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:37:45.390823 5528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781879837648036-392-0/minicluster-data/master-0/data/instance:
uuid: "4441694b3e114df890272f43cf24a406"
format_stamp: "Formatted at 2026-06-19 14:37:45 on dist-test-slave-32hj"
server_key: "abd177fdc92887566bf24f14d026a066"
server_key_iv: "d733963d4a15603ef6d699aece2ec5eb"
server_key_version: "encryptionkey@0"
I20260619 14:37:45.391345 5528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781879837648036-392-0/minicluster-data/master-0/wal/instance:
uuid: "4441694b3e114df890272f43cf24a406"
format_stamp: "Formatted at 2026-06-19 14:37:45 on dist-test-slave-32hj"
server_key: "abd177fdc92887566bf24f14d026a066"
server_key_iv: "d733963d4a15603ef6d699aece2ec5eb"
server_key_version: "encryptionkey@0"
I20260619 14:37:45.394891 5528 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:37:45.397233 5542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:45.398387 5528 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:37:45.398491 5528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781879837648036-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781879837648036-392-0/minicluster-data/master-0/wal
uuid: "4441694b3e114df890272f43cf24a406"
format_stamp: "Formatted at 2026-06-19 14:37:45 on dist-test-slave-32hj"
server_key: "abd177fdc92887566bf24f14d026a066"
server_key_iv: "d733963d4a15603ef6d699aece2ec5eb"
server_key_version: "encryptionkey@0"
I20260619 14:37:45.398578 5528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781879837648036-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781879837648036-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781879837648036-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:37:45.410621 5528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:37:45.411285 5528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:37:45.411453 5528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:37:45.417904 5528 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41249
I20260619 14:37:45.417968 5594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41249 every 8 connection(s)
I20260619 14:37:45.418794 5528 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:37:45.421805 5595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:45.423868 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 5528
I20260619 14:37:45.423991 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781879837648036-392-0/minicluster-data/master-0/wal/instance
I20260619 14:37:45.424266 392 external_mini_cluster.cc:1428] Setting key 81fb5dd7e302ad7c41d8653efa0c8a4c
I20260619 14:37:45.427559 5595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4441694b3e114df890272f43cf24a406: Bootstrap starting.
I20260619 14:37:45.429644 5595 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4441694b3e114df890272f43cf24a406: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:45.430315 5595 log.cc:824] T 00000000000000000000000000000000 P 4441694b3e114df890272f43cf24a406: Log is configured to *not* fsync() on all Append() calls
I20260619 14:37:45.432018 5595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4441694b3e114df890272f43cf24a406: No bootstrap required, opened a new log
I20260619 14:37:45.432713 5598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:45.425591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52380 (local address 127.0.98.62:41249)
0619 14:37:45.426189 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:45.426203 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:45.426237 (+ 34us) server_negotiation.cc:407] Connection header received
0619 14:37:45.426855 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:45.426871 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:45.427250 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:45.427548 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:45.427958 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:45.428924 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:45.429756 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:45.430229 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:45.430382 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:45.431112 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:45.431117 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:45.431129 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:37:45.431493 (+ 364us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:45.431545 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:37:45.431564 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:45.431645 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:45.431766 (+ 121us) server_negotiation.cc:299] Negotiation successful
0619 14:37:45.432009 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":123,"threads_started":1}
I20260619 14:37:45.435483 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:37:45.435741 5595 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4441694b3e114df890272f43cf24a406 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4441694b3e114df890272f43cf24a406" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41249 } }
I20260619 14:37:45.435950 5595 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4441694b3e114df890272f43cf24a406 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:45.436003 5595 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4441694b3e114df890272f43cf24a406 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4441694b3e114df890272f43cf24a406, State: Initialized, Role: FOLLOWER
I20260619 14:37:45.436415 5595 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4441694b3e114df890272f43cf24a406 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4441694b3e114df890272f43cf24a406" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41249 } }
I20260619 14:37:45.436555 5595 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4441694b3e114df890272f43cf24a406 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:37:45.436616 5595 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4441694b3e114df890272f43cf24a406 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:37:45.436707 5595 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4441694b3e114df890272f43cf24a406 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:45.437605 5595 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4441694b3e114df890272f43cf24a406 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4441694b3e114df890272f43cf24a406" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41249 } }
I20260619 14:37:45.437906 5595 leader_election.cc:302] T 00000000000000000000000000000000 P 4441694b3e114df890272f43cf24a406 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4441694b3e114df890272f43cf24a406; no voters:
I20260619 14:37:45.438161 5595 leader_election.cc:288] T 00000000000000000000000000000000 P 4441694b3e114df890272f43cf24a406 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:37:45.438257 5600 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4441694b3e114df890272f43cf24a406 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:37:45.438447 5600 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4441694b3e114df890272f43cf24a406 [term 1 LEADER]: Becoming Leader. State: Replica: 4441694b3e114df890272f43cf24a406, State: Running, Role: LEADER
I20260619 14:37:45.438757 5600 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4441694b3e114df890272f43cf24a406 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4441694b3e114df890272f43cf24a406" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41249 } }
I20260619 14:37:45.439226 5595 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4441694b3e114df890272f43cf24a406 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:37:45.440321 5602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4441694b3e114df890272f43cf24a406 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4441694b3e114df890272f43cf24a406. Latest consensus state: current_term: 1 leader_uuid: "4441694b3e114df890272f43cf24a406" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4441694b3e114df890272f43cf24a406" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41249 } } }
I20260619 14:37:45.440424 5602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4441694b3e114df890272f43cf24a406 [sys.catalog]: This master's current role is: LEADER
I20260619 14:37:45.440766 5609 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:37:45.441079 5601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4441694b3e114df890272f43cf24a406 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4441694b3e114df890272f43cf24a406" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4441694b3e114df890272f43cf24a406" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41249 } } }
I20260619 14:37:45.441169 5601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4441694b3e114df890272f43cf24a406 [sys.catalog]: This master's current role is: LEADER
I20260619 14:37:45.443567 5609 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:37:45.448673 5609 catalog_manager.cc:1347] Generated new cluster ID: 355ba5bf0fc7484680eb215a4d7c7d6e
I20260619 14:37:45.448753 5609 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:37:45.458530 5609 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:37:45.459373 5609 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:37:45.466094 5609 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4441694b3e114df890272f43cf24a406: Generated new TSK 0
I20260619 14:37:45.466681 5609 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 14:37:45.492591 5625 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260619 14:37:45.492710 5625 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 14:37:45.493249 5598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:45.477459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52382 (local address 127.0.98.62:41249)
0619 14:37:45.477568 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:45.477572 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:45.477764 (+ 192us) server_negotiation.cc:407] Connection header received
0619 14:37:45.477986 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:45.477988 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:45.478034 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:45.478116 (+ 82us) server_negotiation.cc:226] Negotiated authn=JWT
0619 14:37:45.478362 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:45.478878 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:45.479731 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:45.479951 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:45.480035 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:45.480340 (+ 305us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 14:37:45.492859 (+ 12519us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39513/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 14:37:45.493117 (+ 258us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52382: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39513/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":36}
W20260619 14:37:45.493336 5598 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52382: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39513/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260619 14:37:45.494365 5624 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:37:45.477396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41249 (local address 127.0.0.1:52382)
0619 14:37:45.477691 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:45.477702 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:45.477819 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:45.478141 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:45.478144 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:45.478152 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0619 14:37:45.478269 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:45.478274 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:45.478980 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:45.478983 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:45.479624 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:45.479632 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:45.480034 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:45.480052 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:45.480143 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:45.480182 (+ 39us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 14:37:45.493068 (+ 12886us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39513/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 14:37:45.493275 (+ 207us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:41249: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39513/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":106,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:41249: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39513/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260619 14:37:45.495092 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 5528
2026-06-19T14:37:45Z chronyd exiting
I20260619 14:37:45.531200 392 test_util.cc:175] -----------------------------------------------
I20260619 14:37:45.531304 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781879837648036-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (284 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260619 14:37:45.532285 392 mini_oidc.cc:152] Starting JWKS server
I20260619 14:37:45.533151 392 webserver.cc:460] Webserver started at https://127.0.0.1:42411/ using document root <none> and password file <none>
I20260619 14:37:45.533231 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 14:37:45.533438 392 webserver.cc:460] Webserver started at http://127.0.0.1:41225/ using document root <none> and password file <none>
2026-06-19T14:37:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:37:45Z Disabled control of system clock
I20260619 14:37:45.541320 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781879837648036-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781879837648036-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781879837648036-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42449
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42449
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42411/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781879837648036-392-0/testchainca.cert with env {}
W20260619 14:37:45.644775 5631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:37:45.645028 5631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:37:45.645066 5631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:37:45.648937 5631 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:37:45.648981 5631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:37:45.648995 5631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:37:45.649009 5631 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:37:45.649022 5631 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 14:37:45.649039 5631 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 14:37:45.649053 5631 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42411/jwks.json
I20260619 14:37:45.654143 5631 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781879837648036-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781879837648036-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42449
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42411/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781879837648036-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781879837648036-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:37:45.655076 5631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:37:45.655978 5631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:37:45.663509 5637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:45.663596 5631 server_base.cc:1034] running on GCE node
W20260619 14:37:45.663604 5636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:45.663513 5639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:45.664166 5631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:37:45.665238 5631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:37:45.666402 5631 hybrid_clock.cc:648] HybridClock initialized: now 1781879865666382 us; error 32 us; skew 500 ppm
I20260619 14:37:45.668218 5631 webserver.cc:460] Webserver started at http://127.0.98.62:40533/ using document root <none> and password file <none>
I20260619 14:37:45.668784 5631 fs_manager.cc:359] Metadata directory not provided
I20260619 14:37:45.668857 5631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:37:45.669050 5631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:37:45.670966 5631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781879837648036-392-0/minicluster-data/master-0/data/instance:
uuid: "edfdcbc79a414867978d87faa1cd02f7"
format_stamp: "Formatted at 2026-06-19 14:37:45 on dist-test-slave-32hj"
server_key: "76519d55d4efdc3adea557873e79b9b2"
server_key_iv: "967a90f6cf95ba831ab3cf0a2c294c1f"
server_key_version: "encryptionkey@0"
I20260619 14:37:45.671507 5631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781879837648036-392-0/minicluster-data/master-0/wal/instance:
uuid: "edfdcbc79a414867978d87faa1cd02f7"
format_stamp: "Formatted at 2026-06-19 14:37:45 on dist-test-slave-32hj"
server_key: "76519d55d4efdc3adea557873e79b9b2"
server_key_iv: "967a90f6cf95ba831ab3cf0a2c294c1f"
server_key_version: "encryptionkey@0"
I20260619 14:37:45.675052 5631 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:37:45.677372 5645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:45.678525 5631 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:37:45.678632 5631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781879837648036-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781879837648036-392-0/minicluster-data/master-0/wal
uuid: "edfdcbc79a414867978d87faa1cd02f7"
format_stamp: "Formatted at 2026-06-19 14:37:45 on dist-test-slave-32hj"
server_key: "76519d55d4efdc3adea557873e79b9b2"
server_key_iv: "967a90f6cf95ba831ab3cf0a2c294c1f"
server_key_version: "encryptionkey@0"
I20260619 14:37:45.678721 5631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781879837648036-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781879837648036-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781879837648036-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:37:45.716337 5631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:37:45.717077 5631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:37:45.717247 5631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:37:45.723517 5631 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42449
I20260619 14:37:45.723529 5697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42449 every 8 connection(s)
I20260619 14:37:45.724401 5631 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:37:45.726140 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 5631
I20260619 14:37:45.726262 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781879837648036-392-0/minicluster-data/master-0/wal/instance
I20260619 14:37:45.726572 392 external_mini_cluster.cc:1428] Setting key 5c7bb77ffec5f610f48f7dad14539398
I20260619 14:37:45.728479 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:45.733855 5698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edfdcbc79a414867978d87faa1cd02f7: Bootstrap starting.
I20260619 14:37:45.735256 5701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:45.727740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54652 (local address 127.0.98.62:42449)
0619 14:37:45.728316 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:45.728330 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:45.728365 (+ 35us) server_negotiation.cc:407] Connection header received
0619 14:37:45.729062 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:45.729081 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:45.729479 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:45.729787 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:45.730196 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:45.731245 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:45.732110 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:45.732580 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:45.732754 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:45.733612 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:45.733617 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:45.733647 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:37:45.734036 (+ 389us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:45.734091 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:37:45.734111 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:45.734192 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:45.734326 (+ 134us) server_negotiation.cc:299] Negotiation successful
0619 14:37:45.734584 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":113,"threads_started":1}
I20260619 14:37:45.736019 5698 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P edfdcbc79a414867978d87faa1cd02f7: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:45.736789 5698 log.cc:824] T 00000000000000000000000000000000 P edfdcbc79a414867978d87faa1cd02f7: Log is configured to *not* fsync() on all Append() calls
I20260619 14:37:45.737836 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:37:45.738631 5698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edfdcbc79a414867978d87faa1cd02f7: No bootstrap required, opened a new log
I20260619 14:37:45.742066 5698 raft_consensus.cc:348] T 00000000000000000000000000000000 P edfdcbc79a414867978d87faa1cd02f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edfdcbc79a414867978d87faa1cd02f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42449 } }
I20260619 14:37:45.742269 5698 raft_consensus.cc:374] T 00000000000000000000000000000000 P edfdcbc79a414867978d87faa1cd02f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:45.742308 5698 raft_consensus.cc:729] T 00000000000000000000000000000000 P edfdcbc79a414867978d87faa1cd02f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edfdcbc79a414867978d87faa1cd02f7, State: Initialized, Role: FOLLOWER
I20260619 14:37:45.742646 5698 consensus_queue.cc:260] T 00000000000000000000000000000000 P edfdcbc79a414867978d87faa1cd02f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edfdcbc79a414867978d87faa1cd02f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42449 } }
I20260619 14:37:45.742749 5698 raft_consensus.cc:388] T 00000000000000000000000000000000 P edfdcbc79a414867978d87faa1cd02f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:37:45.742791 5698 raft_consensus.cc:482] T 00000000000000000000000000000000 P edfdcbc79a414867978d87faa1cd02f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:37:45.742861 5698 raft_consensus.cc:3037] T 00000000000000000000000000000000 P edfdcbc79a414867978d87faa1cd02f7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:45.743641 5698 raft_consensus.cc:504] T 00000000000000000000000000000000 P edfdcbc79a414867978d87faa1cd02f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edfdcbc79a414867978d87faa1cd02f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42449 } }
I20260619 14:37:45.743921 5698 leader_election.cc:302] T 00000000000000000000000000000000 P edfdcbc79a414867978d87faa1cd02f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: edfdcbc79a414867978d87faa1cd02f7; no voters:
I20260619 14:37:45.744176 5698 leader_election.cc:288] T 00000000000000000000000000000000 P edfdcbc79a414867978d87faa1cd02f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:37:45.744285 5703 raft_consensus.cc:2781] T 00000000000000000000000000000000 P edfdcbc79a414867978d87faa1cd02f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:37:45.744496 5703 raft_consensus.cc:686] T 00000000000000000000000000000000 P edfdcbc79a414867978d87faa1cd02f7 [term 1 LEADER]: Becoming Leader. State: Replica: edfdcbc79a414867978d87faa1cd02f7, State: Running, Role: LEADER
I20260619 14:37:45.744812 5703 consensus_queue.cc:237] T 00000000000000000000000000000000 P edfdcbc79a414867978d87faa1cd02f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edfdcbc79a414867978d87faa1cd02f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42449 } }
I20260619 14:37:45.745239 5698 sys_catalog.cc:564] T 00000000000000000000000000000000 P edfdcbc79a414867978d87faa1cd02f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:37:45.746384 5705 sys_catalog.cc:455] T 00000000000000000000000000000000 P edfdcbc79a414867978d87faa1cd02f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader edfdcbc79a414867978d87faa1cd02f7. Latest consensus state: current_term: 1 leader_uuid: "edfdcbc79a414867978d87faa1cd02f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edfdcbc79a414867978d87faa1cd02f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42449 } } }
I20260619 14:37:45.746483 5705 sys_catalog.cc:458] T 00000000000000000000000000000000 P edfdcbc79a414867978d87faa1cd02f7 [sys.catalog]: This master's current role is: LEADER
I20260619 14:37:45.746425 5704 sys_catalog.cc:455] T 00000000000000000000000000000000 P edfdcbc79a414867978d87faa1cd02f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "edfdcbc79a414867978d87faa1cd02f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edfdcbc79a414867978d87faa1cd02f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42449 } } }
I20260619 14:37:45.746546 5704 sys_catalog.cc:458] T 00000000000000000000000000000000 P edfdcbc79a414867978d87faa1cd02f7 [sys.catalog]: This master's current role is: LEADER
I20260619 14:37:45.746850 5712 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:37:45.749742 5712 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:37:45.754576 5712 catalog_manager.cc:1347] Generated new cluster ID: de1bb7c2a01a4611b3a764a9eabbbb5c
I20260619 14:37:45.754640 5712 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:37:45.763110 5712 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:37:45.763924 5712 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:37:45.777545 5712 catalog_manager.cc:5868] T 00000000000000000000000000000000 P edfdcbc79a414867978d87faa1cd02f7: Generated new TSK 0
I20260619 14:37:45.778303 5712 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 14:37:45.811177 5728 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260619 14:37:45.811277 5728 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 14:37:45.811857 5701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:45.795427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54668 (local address 127.0.98.62:42449)
0619 14:37:45.795562 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:45.795567 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:45.795753 (+ 186us) server_negotiation.cc:407] Connection header received
0619 14:37:45.795937 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:45.795946 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:45.796016 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:45.796108 (+ 92us) server_negotiation.cc:226] Negotiated authn=JWT
0619 14:37:45.796393 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:45.797038 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:45.798322 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:45.798531 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:45.798611 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:45.798926 (+ 315us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 14:37:45.811531 (+ 12605us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42411/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 14:37:45.811737 (+ 206us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54668: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42411/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":52}
W20260619 14:37:45.811916 5701 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54668: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42411/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260619 14:37:45.812094 5727 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:37:45.795344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42449 (local address 127.0.0.1:54668)
0619 14:37:45.795644 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:45.795659 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:45.795775 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:45.796131 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:45.796135 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:45.796146 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0619 14:37:45.796266 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:45.796272 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:45.797154 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:45.797157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:45.798200 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:45.798211 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:45.798617 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:45.798638 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:45.798726 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:45.798753 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 14:37:45.811725 (+ 12972us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42411/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 14:37:45.811901 (+ 176us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:42449: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42411/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":94,"threads_started":1}
I20260619 14:37:45.812645 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 5631
2026-06-19T14:37:45Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (317 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260619 14:37:45.850232 392 mini_oidc.cc:152] Starting JWKS server
I20260619 14:37:45.851028 392 webserver.cc:460] Webserver started at https://127.0.0.1:33749/ using document root <none> and password file <none>
I20260619 14:37:45.851120 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 14:37:45.851361 392 webserver.cc:460] Webserver started at http://127.0.0.1:33291/ using document root <none> and password file <none>
2026-06-19T14:37:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:37:45Z Disabled control of system clock
I20260619 14:37:45.858793 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781879837648036-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781879837648036-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781879837648036-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35867
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35867
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33749/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 14:37:45.962028 5734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:37:45.962296 5734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:37:45.962333 5734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:37:45.966658 5734 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:37:45.966727 5734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:37:45.966740 5734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:37:45.966756 5734 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:37:45.966770 5734 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 14:37:45.966784 5734 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 14:37:45.966799 5734 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33749/jwks.json
I20260619 14:37:45.971539 5734 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781879837648036-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781879837648036-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35867
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33749/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781879837648036-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:37:45.972632 5734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:37:45.973977 5734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:37:45.982028 5742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:45.982028 5740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:45.982020 5739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:45.982369 5734 server_base.cc:1034] running on GCE node
I20260619 14:37:45.982903 5734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:37:45.983932 5734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:37:45.985085 5734 hybrid_clock.cc:648] HybridClock initialized: now 1781879865985057 us; error 48 us; skew 500 ppm
I20260619 14:37:45.986824 5734 webserver.cc:460] Webserver started at http://127.0.98.62:40639/ using document root <none> and password file <none>
I20260619 14:37:45.987349 5734 fs_manager.cc:359] Metadata directory not provided
I20260619 14:37:45.987413 5734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:37:45.987591 5734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:37:45.989596 5734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781879837648036-392-0/minicluster-data/master-0/data/instance:
uuid: "30d8adbfc0fd4499bde3e6d459ee27fb"
format_stamp: "Formatted at 2026-06-19 14:37:45 on dist-test-slave-32hj"
server_key: "6b2b7df6d8cb7e6a9cad127bab9011a4"
server_key_iv: "e76fd47741666b96cce84060df9b3f60"
server_key_version: "encryptionkey@0"
I20260619 14:37:45.990118 5734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781879837648036-392-0/minicluster-data/master-0/wal/instance:
uuid: "30d8adbfc0fd4499bde3e6d459ee27fb"
format_stamp: "Formatted at 2026-06-19 14:37:45 on dist-test-slave-32hj"
server_key: "6b2b7df6d8cb7e6a9cad127bab9011a4"
server_key_iv: "e76fd47741666b96cce84060df9b3f60"
server_key_version: "encryptionkey@0"
I20260619 14:37:45.993615 5734 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260619 14:37:45.995977 5748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:45.997169 5734 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 14:37:45.997277 5734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781879837648036-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781879837648036-392-0/minicluster-data/master-0/wal
uuid: "30d8adbfc0fd4499bde3e6d459ee27fb"
format_stamp: "Formatted at 2026-06-19 14:37:45 on dist-test-slave-32hj"
server_key: "6b2b7df6d8cb7e6a9cad127bab9011a4"
server_key_iv: "e76fd47741666b96cce84060df9b3f60"
server_key_version: "encryptionkey@0"
I20260619 14:37:45.997368 5734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781879837648036-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781879837648036-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781879837648036-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:37:46.014205 5734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:37:46.014891 5734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:37:46.015061 5734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:37:46.021510 5734 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35867
I20260619 14:37:46.021566 5800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35867 every 8 connection(s)
I20260619 14:37:46.022432 5734 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:37:46.023360 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 5734
I20260619 14:37:46.023475 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781879837648036-392-0/minicluster-data/master-0/wal/instance
I20260619 14:37:46.023730 392 external_mini_cluster.cc:1428] Setting key 410157dcf2e15440b687385181ba3b8e
I20260619 14:37:46.025827 5801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:46.031114 5801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30d8adbfc0fd4499bde3e6d459ee27fb: Bootstrap starting.
I20260619 14:37:46.032456 5804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:46.025055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51658 (local address 127.0.98.62:35867)
0619 14:37:46.025682 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:46.025697 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:46.025732 (+ 35us) server_negotiation.cc:407] Connection header received
0619 14:37:46.026382 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:46.026400 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:46.026788 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:46.027112 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:46.027517 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:46.028492 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:46.029390 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:46.029843 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:46.029992 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:46.030765 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:46.030770 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:46.030781 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:37:46.031154 (+ 373us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:46.031208 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:37:46.031227 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:46.031313 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:46.031483 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 14:37:46.031744 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":139,"threads_started":1}
I20260619 14:37:46.033382 5801 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30d8adbfc0fd4499bde3e6d459ee27fb: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:46.034206 5801 log.cc:824] T 00000000000000000000000000000000 P 30d8adbfc0fd4499bde3e6d459ee27fb: Log is configured to *not* fsync() on all Append() calls
I20260619 14:37:46.035183 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:37:46.036278 5801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30d8adbfc0fd4499bde3e6d459ee27fb: No bootstrap required, opened a new log
I20260619 14:37:46.039676 5801 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30d8adbfc0fd4499bde3e6d459ee27fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d8adbfc0fd4499bde3e6d459ee27fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35867 } }
I20260619 14:37:46.039883 5801 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30d8adbfc0fd4499bde3e6d459ee27fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:46.039945 5801 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30d8adbfc0fd4499bde3e6d459ee27fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30d8adbfc0fd4499bde3e6d459ee27fb, State: Initialized, Role: FOLLOWER
I20260619 14:37:46.040346 5801 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30d8adbfc0fd4499bde3e6d459ee27fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d8adbfc0fd4499bde3e6d459ee27fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35867 } }
I20260619 14:37:46.040474 5801 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30d8adbfc0fd4499bde3e6d459ee27fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:37:46.040535 5801 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30d8adbfc0fd4499bde3e6d459ee27fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:37:46.040627 5801 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30d8adbfc0fd4499bde3e6d459ee27fb [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:46.041566 5801 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30d8adbfc0fd4499bde3e6d459ee27fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d8adbfc0fd4499bde3e6d459ee27fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35867 } }
I20260619 14:37:46.041915 5801 leader_election.cc:302] T 00000000000000000000000000000000 P 30d8adbfc0fd4499bde3e6d459ee27fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30d8adbfc0fd4499bde3e6d459ee27fb; no voters:
I20260619 14:37:46.042183 5801 leader_election.cc:288] T 00000000000000000000000000000000 P 30d8adbfc0fd4499bde3e6d459ee27fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:37:46.042277 5806 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30d8adbfc0fd4499bde3e6d459ee27fb [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:37:46.042465 5806 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30d8adbfc0fd4499bde3e6d459ee27fb [term 1 LEADER]: Becoming Leader. State: Replica: 30d8adbfc0fd4499bde3e6d459ee27fb, State: Running, Role: LEADER
I20260619 14:37:46.042747 5806 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30d8adbfc0fd4499bde3e6d459ee27fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d8adbfc0fd4499bde3e6d459ee27fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35867 } }
I20260619 14:37:46.043278 5801 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30d8adbfc0fd4499bde3e6d459ee27fb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:37:46.044204 5808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30d8adbfc0fd4499bde3e6d459ee27fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30d8adbfc0fd4499bde3e6d459ee27fb. Latest consensus state: current_term: 1 leader_uuid: "30d8adbfc0fd4499bde3e6d459ee27fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d8adbfc0fd4499bde3e6d459ee27fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35867 } } }
I20260619 14:37:46.044333 5808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30d8adbfc0fd4499bde3e6d459ee27fb [sys.catalog]: This master's current role is: LEADER
I20260619 14:37:46.044705 5815 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:37:46.045140 5807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30d8adbfc0fd4499bde3e6d459ee27fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30d8adbfc0fd4499bde3e6d459ee27fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d8adbfc0fd4499bde3e6d459ee27fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35867 } } }
I20260619 14:37:46.045233 5807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30d8adbfc0fd4499bde3e6d459ee27fb [sys.catalog]: This master's current role is: LEADER
I20260619 14:37:46.047776 5815 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:37:46.053117 5815 catalog_manager.cc:1347] Generated new cluster ID: eaca80f3e11847ff844fec9282638991
I20260619 14:37:46.053179 5815 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:37:46.078411 5815 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:37:46.079319 5815 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:37:46.085814 5815 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30d8adbfc0fd4499bde3e6d459ee27fb: Generated new TSK 0
I20260619 14:37:46.086634 5815 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:37:46.111050 5804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:46.106199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51668 (local address 127.0.98.62:35867)
0619 14:37:46.106321 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:46.106325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:46.106457 (+ 132us) server_negotiation.cc:407] Connection header received
0619 14:37:46.106593 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:46.106596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:46.106649 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:46.106764 (+ 115us) server_negotiation.cc:226] Negotiated authn=JWT
0619 14:37:46.106915 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:46.107713 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:46.108531 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:46.108894 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:46.108995 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:46.109267 (+ 272us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 14:37:46.110695 (+ 1428us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33749/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 14:37:46.110917 (+ 222us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51668: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33749/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":45}
W20260619 14:37:46.111119 5804 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51668: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33749/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260619 14:37:46.112665 5830 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:37:46.106109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35867 (local address 127.0.0.1:51668)
0619 14:37:46.106381 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:46.106393 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:46.106488 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:46.106753 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:46.106755 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:46.106760 (+ 5us) client_negotiation.cc:189] Negotiated authn=JWT
0619 14:37:46.106847 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:46.106851 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:46.107802 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:46.107804 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:46.108433 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:46.108440 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:46.108988 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:46.109009 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:46.109074 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:46.109097 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 14:37:46.110853 (+ 1756us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33749/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 14:37:46.112504 (+ 1651us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:35867: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33749/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":1400,"thread_start_us":101,"threads_started":1}
I20260619 14:37:46.113304 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 5734
E20260619 14:37:46.121801 5831 webserver.cc:533] Webserver: sslize failed:
E20260619 14:37:46.121871 5831 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-19T14:37:46Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (300 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-19T14:37:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:37:46Z Disabled control of system clock
I20260619 14:37:46.158303 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781879837648036-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781879837648036-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781879837648036-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41123
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:32871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41123
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781879837648036-392-0/testchainca.cert with env {}
W20260619 14:37:46.261320 5835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:37:46.261570 5835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:37:46.261607 5835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:37:46.265571 5835 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:37:46.265622 5835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:37:46.265635 5835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:37:46.265650 5835 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:37:46.265676 5835 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 14:37:46.265702 5835 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 14:37:46.265726 5835 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260619 14:37:46.270831 5835 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781879837648036-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781879837648036-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41123
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781879837648036-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781879837648036-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:37:46.271749 5835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:37:46.272707 5835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:37:46.280411 5841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:46.280383 5840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:46.280498 5835 server_base.cc:1034] running on GCE node
W20260619 14:37:46.280393 5843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:46.281159 5835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:37:46.282147 5835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:37:46.283301 5835 hybrid_clock.cc:648] HybridClock initialized: now 1781879866283285 us; error 35 us; skew 500 ppm
I20260619 14:37:46.285061 5835 webserver.cc:460] Webserver started at http://127.0.98.62:41485/ using document root <none> and password file <none>
I20260619 14:37:46.285588 5835 fs_manager.cc:359] Metadata directory not provided
I20260619 14:37:46.285651 5835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:37:46.285826 5835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:37:46.287734 5835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781879837648036-392-0/minicluster-data/master-0/data/instance:
uuid: "c5dda573896e423a9aaef7bf4efc50b6"
format_stamp: "Formatted at 2026-06-19 14:37:46 on dist-test-slave-32hj"
server_key: "b71696aba461c5f0a418bec322cae7b7"
server_key_iv: "2760669c4487076e63f8cf2242018c55"
server_key_version: "encryptionkey@0"
I20260619 14:37:46.288241 5835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781879837648036-392-0/minicluster-data/master-0/wal/instance:
uuid: "c5dda573896e423a9aaef7bf4efc50b6"
format_stamp: "Formatted at 2026-06-19 14:37:46 on dist-test-slave-32hj"
server_key: "b71696aba461c5f0a418bec322cae7b7"
server_key_iv: "2760669c4487076e63f8cf2242018c55"
server_key_version: "encryptionkey@0"
I20260619 14:37:46.291643 5835 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 14:37:46.293956 5849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:46.295135 5835 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:37:46.295240 5835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781879837648036-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781879837648036-392-0/minicluster-data/master-0/wal
uuid: "c5dda573896e423a9aaef7bf4efc50b6"
format_stamp: "Formatted at 2026-06-19 14:37:46 on dist-test-slave-32hj"
server_key: "b71696aba461c5f0a418bec322cae7b7"
server_key_iv: "2760669c4487076e63f8cf2242018c55"
server_key_version: "encryptionkey@0"
I20260619 14:37:46.295334 5835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781879837648036-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781879837648036-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781879837648036-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:37:46.316622 5835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:37:46.317286 5835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:37:46.317456 5835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:37:46.323596 5835 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41123
I20260619 14:37:46.323614 5901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41123 every 8 connection(s)
I20260619 14:37:46.324455 5835 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:37:46.327093 5902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:46.332159 5902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5dda573896e423a9aaef7bf4efc50b6: Bootstrap starting.
I20260619 14:37:46.333178 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 5835
I20260619 14:37:46.333308 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781879837648036-392-0/minicluster-data/master-0/wal/instance
I20260619 14:37:46.333586 392 external_mini_cluster.cc:1428] Setting key 9d3cbc818e4befda8e3294e908e0cd9d
I20260619 14:37:46.334786 5902 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5dda573896e423a9aaef7bf4efc50b6: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:46.335471 5902 log.cc:824] T 00000000000000000000000000000000 P c5dda573896e423a9aaef7bf4efc50b6: Log is configured to *not* fsync() on all Append() calls
I20260619 14:37:46.337306 5902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5dda573896e423a9aaef7bf4efc50b6: No bootstrap required, opened a new log
I20260619 14:37:46.340466 5902 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5dda573896e423a9aaef7bf4efc50b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5dda573896e423a9aaef7bf4efc50b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41123 } }
I20260619 14:37:46.340662 5902 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5dda573896e423a9aaef7bf4efc50b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:46.340713 5902 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5dda573896e423a9aaef7bf4efc50b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5dda573896e423a9aaef7bf4efc50b6, State: Initialized, Role: FOLLOWER
I20260619 14:37:46.341198 5902 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5dda573896e423a9aaef7bf4efc50b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5dda573896e423a9aaef7bf4efc50b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41123 } }
I20260619 14:37:46.341334 5902 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5dda573896e423a9aaef7bf4efc50b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:37:46.341398 5902 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5dda573896e423a9aaef7bf4efc50b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:37:46.341501 5902 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5dda573896e423a9aaef7bf4efc50b6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:46.341902 5905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:46.334583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53366 (local address 127.0.98.62:41123)
0619 14:37:46.335201 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:46.335215 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:46.335249 (+ 34us) server_negotiation.cc:407] Connection header received
0619 14:37:46.335876 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:46.335899 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:46.336297 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:46.336596 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:46.337027 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:46.337972 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:46.338809 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:46.339275 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:46.339416 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:46.340172 (+ 756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:46.340177 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:46.340189 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:37:46.340546 (+ 357us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:46.340601 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:37:46.340619 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:46.340709 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:46.340903 (+ 194us) server_negotiation.cc:299] Negotiation successful
0619 14:37:46.341143 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":120,"threads_started":1}
I20260619 14:37:46.342363 5902 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5dda573896e423a9aaef7bf4efc50b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5dda573896e423a9aaef7bf4efc50b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41123 } }
I20260619 14:37:46.342674 5902 leader_election.cc:302] T 00000000000000000000000000000000 P c5dda573896e423a9aaef7bf4efc50b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5dda573896e423a9aaef7bf4efc50b6; no voters:
I20260619 14:37:46.342984 5902 leader_election.cc:288] T 00000000000000000000000000000000 P c5dda573896e423a9aaef7bf4efc50b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:37:46.343067 5907 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5dda573896e423a9aaef7bf4efc50b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:37:46.343261 5907 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5dda573896e423a9aaef7bf4efc50b6 [term 1 LEADER]: Becoming Leader. State: Replica: c5dda573896e423a9aaef7bf4efc50b6, State: Running, Role: LEADER
I20260619 14:37:46.343577 5907 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5dda573896e423a9aaef7bf4efc50b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5dda573896e423a9aaef7bf4efc50b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41123 } }
I20260619 14:37:46.344185 5902 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5dda573896e423a9aaef7bf4efc50b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:37:46.344683 5909 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5dda573896e423a9aaef7bf4efc50b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5dda573896e423a9aaef7bf4efc50b6. Latest consensus state: current_term: 1 leader_uuid: "c5dda573896e423a9aaef7bf4efc50b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5dda573896e423a9aaef7bf4efc50b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41123 } } }
I20260619 14:37:46.344759 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:37:46.344821 5909 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5dda573896e423a9aaef7bf4efc50b6 [sys.catalog]: This master's current role is: LEADER
I20260619 14:37:46.346032 5908 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5dda573896e423a9aaef7bf4efc50b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5dda573896e423a9aaef7bf4efc50b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5dda573896e423a9aaef7bf4efc50b6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41123 } } }
I20260619 14:37:46.346117 5908 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5dda573896e423a9aaef7bf4efc50b6 [sys.catalog]: This master's current role is: LEADER
W20260619 14:37:46.348672 5922 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c5dda573896e423a9aaef7bf4efc50b6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:37:46.348758 5922 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:37:46.348824 5912 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:37:46.349484 5912 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:37:46.354290 5912 catalog_manager.cc:1347] Generated new cluster ID: 6e499ade39c349419a8dad94b3a6d8f0
I20260619 14:37:46.354355 5912 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:37:46.373255 5912 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:37:46.374173 5912 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:37:46.380337 5912 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5dda573896e423a9aaef7bf4efc50b6: Generated new TSK 0
I20260619 14:37:46.381349 5912 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:37:46.895532 5905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:46.394058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53374 (local address 127.0.98.62:41123)
0619 14:37:46.394171 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:46.394176 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:46.394297 (+ 121us) server_negotiation.cc:407] Connection header received
0619 14:37:46.394439 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:46.394442 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:46.394496 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:46.394605 (+ 109us) server_negotiation.cc:226] Negotiated authn=JWT
0619 14:37:46.394759 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:46.395591 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:46.396451 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:46.396823 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:46.396920 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:46.397154 (+ 234us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 14:37:46.895145 (+497991us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 14:37:46.895413 (+ 268us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53374: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":40}
W20260619 14:37:46.895601 5905 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53374: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260619 14:37:46.895726 5931 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:37:46.393989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41123 (local address 127.0.0.1:53374)
0619 14:37:46.394227 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:46.394238 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:46.394324 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:46.394595 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:46.394597 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:46.394603 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0619 14:37:46.394684 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:46.394688 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:46.395709 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:46.395712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:46.396343 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:46.396351 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:46.396914 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:46.396929 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:46.396990 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:46.397010 (+ 20us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 14:37:46.895361 (+498351us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 14:37:46.895535 (+ 174us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:41123: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":93,"threads_started":1}
I20260619 14:37:46.896255 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 5835
2026-06-19T14:37:46Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (763 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (115 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (116 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:37:47 dist-test-slave-32hj krb5kdc[5943](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:37:47 dist-test-slave-32hj krb5kdc[5943](info): set up 2 sockets
Jun 19 14:37:47 dist-test-slave-32hj krb5kdc[5943](info): commencing operation
krb5kdc: starting...
W20260619 14:37:48.076776 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/krb5kdc/test-user.keytab.
Jun 19 14:37:48 dist-test-slave-32hj krb5kdc[5943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:37:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:37:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:37:48.189843 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33905
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33905
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:37:48.295009 5956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:37:48.295295 5956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:37:48.295351 5956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:37:48.299650 5956 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:37:48.299741 5956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:37:48.299774 5956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:37:48.299794 5956 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:37:48.299815 5956 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:37:48.305330 5956 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33905
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:37:48.306392 5956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:37:48.307447 5956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:37:48.315393 5961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:48.315460 5962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:48.315569 5964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:48.315618 5956 server_base.cc:1034] running on GCE node
I20260619 14:37:48.316179 5956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:37:48.317268 5956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:37:48.318416 5956 hybrid_clock.cc:648] HybridClock initialized: now 1781879868318387 us; error 44 us; skew 500 ppm
Jun 19 14:37:48 dist-test-slave-32hj krb5kdc[5943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879868, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:37:48.320992 5956 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:37:48.321771 5956 webserver.cc:460] Webserver started at http://127.0.98.62:43183/ using document root <none> and password file <none>
I20260619 14:37:48.322263 5956 fs_manager.cc:359] Metadata directory not provided
I20260619 14:37:48.322315 5956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:37:48.322484 5956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:37:48.324357 5956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/master-0/data/instance:
uuid: "48a5d6d5c6f949f095305f371a77aeac"
format_stamp: "Formatted at 2026-06-19 14:37:48 on dist-test-slave-32hj"
server_key: "10fbbbe6d8c35309f5e9d925bd7117d8"
server_key_iv: "29675825a866900ffec219a9f7a1e7fb"
server_key_version: "encryptionkey@0"
I20260619 14:37:48.324892 5956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/master-0/wal/instance:
uuid: "48a5d6d5c6f949f095305f371a77aeac"
format_stamp: "Formatted at 2026-06-19 14:37:48 on dist-test-slave-32hj"
server_key: "10fbbbe6d8c35309f5e9d925bd7117d8"
server_key_iv: "29675825a866900ffec219a9f7a1e7fb"
server_key_version: "encryptionkey@0"
I20260619 14:37:48.328298 5956 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:37:48.330531 5971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:48.331712 5956 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:37:48.331815 5956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/master-0/wal
uuid: "48a5d6d5c6f949f095305f371a77aeac"
format_stamp: "Formatted at 2026-06-19 14:37:48 on dist-test-slave-32hj"
server_key: "10fbbbe6d8c35309f5e9d925bd7117d8"
server_key_iv: "29675825a866900ffec219a9f7a1e7fb"
server_key_version: "encryptionkey@0"
I20260619 14:37:48.331935 5956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:37:48.341336 5956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:37:48.344288 5956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:37:48.344455 5956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:37:48.350968 5956 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33905
I20260619 14:37:48.350996 6023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33905 every 8 connection(s)
I20260619 14:37:48.351833 5956 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:37:48.354516 6024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:48.355311 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 5956
I20260619 14:37:48.355438 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/master-0/wal/instance
I20260619 14:37:48.355705 392 external_mini_cluster.cc:1428] Setting key 3ad191ccf2e97923dfc3f30f975b3df2
I20260619 14:37:48.360276 6024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48a5d6d5c6f949f095305f371a77aeac: Bootstrap starting.
I20260619 14:37:48.362759 6024 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48a5d6d5c6f949f095305f371a77aeac: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:48.363642 6024 log.cc:824] T 00000000000000000000000000000000 P 48a5d6d5c6f949f095305f371a77aeac: Log is configured to *not* fsync() on all Append() calls
Jun 19 14:37:48 dist-test-slave-32hj krb5kdc[5943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:37:48.365453 6024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48a5d6d5c6f949f095305f371a77aeac: No bootstrap required, opened a new log
I20260619 14:37:48.368640 6027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:48.356946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42226 (local address 127.0.98.62:33905)
0619 14:37:48.357298 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:48.357314 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:48.357347 (+ 33us) server_negotiation.cc:407] Connection header received
0619 14:37:48.357957 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:48.357973 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:48.358284 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:48.358547 (+ 263us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:48.359382 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:48.360320 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:48.361444 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:48.361899 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:48.362011 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:48.364525 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:48.364531 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:48.364543 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:48.364569 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:48.366118 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:48.366509 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:48.366514 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:48.366518 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:48.366622 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:48.366863 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:48.366866 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:48.366868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:48.367233 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:48.367400 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:48.367670 (+ 270us) server_negotiation.cc:299] Negotiation successful
0619 14:37:48.367938 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":90,"threads_started":1}
I20260619 14:37:48.368791 6024 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48a5d6d5c6f949f095305f371a77aeac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48a5d6d5c6f949f095305f371a77aeac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33905 } }
I20260619 14:37:48.369035 6024 raft_consensus.cc:374] T 00000000000000000000000000000000 P 48a5d6d5c6f949f095305f371a77aeac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:48.369098 6024 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48a5d6d5c6f949f095305f371a77aeac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48a5d6d5c6f949f095305f371a77aeac, State: Initialized, Role: FOLLOWER
I20260619 14:37:48.369509 6024 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48a5d6d5c6f949f095305f371a77aeac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48a5d6d5c6f949f095305f371a77aeac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33905 } }
I20260619 14:37:48.369630 6024 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48a5d6d5c6f949f095305f371a77aeac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:37:48.369692 6024 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48a5d6d5c6f949f095305f371a77aeac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:37:48.369786 6024 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48a5d6d5c6f949f095305f371a77aeac [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:48.370625 6024 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48a5d6d5c6f949f095305f371a77aeac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48a5d6d5c6f949f095305f371a77aeac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33905 } }
I20260619 14:37:48.370960 6024 leader_election.cc:302] T 00000000000000000000000000000000 P 48a5d6d5c6f949f095305f371a77aeac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48a5d6d5c6f949f095305f371a77aeac; no voters:
I20260619 14:37:48.371208 6024 leader_election.cc:288] T 00000000000000000000000000000000 P 48a5d6d5c6f949f095305f371a77aeac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:37:48.371304 6029 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48a5d6d5c6f949f095305f371a77aeac [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:37:48.371511 6029 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48a5d6d5c6f949f095305f371a77aeac [term 1 LEADER]: Becoming Leader. State: Replica: 48a5d6d5c6f949f095305f371a77aeac, State: Running, Role: LEADER
I20260619 14:37:48.371784 6029 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48a5d6d5c6f949f095305f371a77aeac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48a5d6d5c6f949f095305f371a77aeac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33905 } }
I20260619 14:37:48.372290 6024 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48a5d6d5c6f949f095305f371a77aeac [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:37:48.373760 6030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48a5d6d5c6f949f095305f371a77aeac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "48a5d6d5c6f949f095305f371a77aeac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48a5d6d5c6f949f095305f371a77aeac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33905 } } }
I20260619 14:37:48.373903 6030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48a5d6d5c6f949f095305f371a77aeac [sys.catalog]: This master's current role is: LEADER
I20260619 14:37:48.374156 6031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48a5d6d5c6f949f095305f371a77aeac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48a5d6d5c6f949f095305f371a77aeac. Latest consensus state: current_term: 1 leader_uuid: "48a5d6d5c6f949f095305f371a77aeac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48a5d6d5c6f949f095305f371a77aeac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33905 } } }
I20260619 14:37:48.374238 6031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48a5d6d5c6f949f095305f371a77aeac [sys.catalog]: This master's current role is: LEADER
I20260619 14:37:48.374670 6042 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:37:48.377303 6042 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:37:48.382403 6042 catalog_manager.cc:1347] Generated new cluster ID: c9f641cf09bf4304978a7b729efc48e3
I20260619 14:37:48.382478 6042 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:37:48.392246 6042 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:37:48.393589 6042 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:37:48.404326 6042 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 48a5d6d5c6f949f095305f371a77aeac: Generated new TSK 0
I20260619 14:37:48.405077 6042 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 14:37:48.434792 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33905
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:37:48.542438 6050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:37:48.542689 6050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:37:48.542747 6050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:37:48.546728 6050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:37:48.546779 6050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:37:48.546850 6050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:37:48.551858 6050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:37:48.552901 6050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:37:48.553857 6050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:37:48.562917 6056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:48.563040 6050 server_base.cc:1034] running on GCE node
W20260619 14:37:48.562899 6055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:48.562917 6058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:48.563544 6050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:37:48.564175 6050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:37:48.565325 6050 hybrid_clock.cc:648] HybridClock initialized: now 1781879868565305 us; error 33 us; skew 500 ppm
Jun 19 14:37:48 dist-test-slave-32hj krb5kdc[5943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879868, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:37:48.567987 6050 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:37:48.568874 6050 webserver.cc:460] Webserver started at http://127.0.98.1:46435/ using document root <none> and password file <none>
I20260619 14:37:48.569407 6050 fs_manager.cc:359] Metadata directory not provided
I20260619 14:37:48.569471 6050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:37:48.569669 6050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:37:48.571661 6050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/ts-0/data/instance:
uuid: "6c74644505a741f3b1b0d1f1bc1d1562"
format_stamp: "Formatted at 2026-06-19 14:37:48 on dist-test-slave-32hj"
server_key: "72a8181b26f41cae58b4635cd0a0efde"
server_key_iv: "776c7e035148db6db7cd8dde4c183aec"
server_key_version: "encryptionkey@0"
I20260619 14:37:48.572201 6050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6c74644505a741f3b1b0d1f1bc1d1562"
format_stamp: "Formatted at 2026-06-19 14:37:48 on dist-test-slave-32hj"
server_key: "72a8181b26f41cae58b4635cd0a0efde"
server_key_iv: "776c7e035148db6db7cd8dde4c183aec"
server_key_version: "encryptionkey@0"
I20260619 14:37:48.575969 6050 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 14:37:48.578269 6065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:48.579527 6050 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:37:48.579641 6050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/ts-0/wal
uuid: "6c74644505a741f3b1b0d1f1bc1d1562"
format_stamp: "Formatted at 2026-06-19 14:37:48 on dist-test-slave-32hj"
server_key: "72a8181b26f41cae58b4635cd0a0efde"
server_key_iv: "776c7e035148db6db7cd8dde4c183aec"
server_key_version: "encryptionkey@0"
I20260619 14:37:48.579737 6050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:37:48.589686 6050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:37:48.592687 6050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:37:48.592895 6050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:37:48.593443 6050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:37:48.594213 6050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:37:48.594259 6050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:48.594309 6050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:37:48.594337 6050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:48.602448 6050 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43877
I20260619 14:37:48.602479 6178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43877 every 8 connection(s)
I20260619 14:37:48.603268 6050 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:37:48.610796 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 6050
I20260619 14:37:48.610913 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:37:48.611219 392 external_mini_cluster.cc:1428] Setting key 588232310cde3684729e4976fa8ac5f4
Jun 19 14:37:48 dist-test-slave-32hj krb5kdc[5943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879868, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:37:48.616933 6027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:48.604916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35787 (local address 127.0.98.62:33905)
0619 14:37:48.605062 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:48.605066 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:48.605981 (+ 915us) server_negotiation.cc:407] Connection header received
0619 14:37:48.607342 (+ 1361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:48.607345 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:48.607393 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:48.607467 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:48.608668 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:48.609238 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:48.610322 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:48.610558 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:48.610664 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:48.613339 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:48.613342 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:48.613345 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:48.613367 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:48.614930 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:48.615422 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:48.615426 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:48.615427 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:48.615513 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:48.615803 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:48.615805 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:48.615807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:48.616017 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:48.616105 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:48.616609 (+ 504us) server_negotiation.cc:299] Negotiation successful
0619 14:37:48.616757 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 14:37:48.617733 6181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:48.605356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33905 (local address 127.0.98.1:35787)
0619 14:37:48.605828 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:48.605871 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:48.607141 (+ 1270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:48.607681 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:48.607690 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:48.608039 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:48.608519 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:48.608533 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:48.609337 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:48.609340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:48.610204 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:48.610212 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:48.610730 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:48.610735 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:48.610898 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:48.611479 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:48.611496 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:48.613163 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:48.615055 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:48.615060 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:48.615073 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:48.615329 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:48.615591 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:48.615593 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:48.615594 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:48.615730 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:48.616112 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:48.616118 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:48.616360 (+ 242us) client_negotiation.cc:769] Sending connection context
0619 14:37:48.616548 (+ 188us) client_negotiation.cc:240] Negotiation successful
0619 14:37:48.616883 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":117,"threads_started":1}
I20260619 14:37:48.618976 6179 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33905
I20260619 14:37:48.619241 6179 heartbeater.cc:464] Registering TS with master...
I20260619 14:37:48.619827 6179 heartbeater.cc:511] Master 127.0.98.62:33905 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 14:37:48.621356 5988 ts_manager.cc:194] Registered new tserver with Master: 6c74644505a741f3b1b0d1f1bc1d1562 (127.0.98.1:43877)
I20260619 14:37:48.622565 5988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35787
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 14:37:48.637090 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33905
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:37:48.743449 6184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:37:48.743698 6184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:37:48.743755 6184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:37:48.747682 6184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:37:48.747733 6184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:37:48.747804 6184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:37:48.752954 6184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:37:48.753947 6184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:37:48.754920 6184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:37:48.763995 6190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:48.764034 6189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:48.764285 6184 server_base.cc:1034] running on GCE node
W20260619 14:37:48.764034 6192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:48.764778 6184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:37:48.765383 6184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:37:48.766548 6184 hybrid_clock.cc:648] HybridClock initialized: now 1781879868766526 us; error 32 us; skew 500 ppm
Jun 19 14:37:48 dist-test-slave-32hj krb5kdc[5943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879868, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:37:48.769235 6184 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:37:48.770133 6184 webserver.cc:460] Webserver started at http://127.0.98.2:33099/ using document root <none> and password file <none>
I20260619 14:37:48.770661 6184 fs_manager.cc:359] Metadata directory not provided
I20260619 14:37:48.770726 6184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:37:48.770928 6184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:37:48.773104 6184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/ts-1/data/instance:
uuid: "5fa8307e6d514deeb7b2a7795ebb3eaf"
format_stamp: "Formatted at 2026-06-19 14:37:48 on dist-test-slave-32hj"
server_key: "fde727016fc37fcbdd3b726a7a1a58cb"
server_key_iv: "c6e9096c72f2941fdb735064ccb1f755"
server_key_version: "encryptionkey@0"
I20260619 14:37:48.773663 6184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5fa8307e6d514deeb7b2a7795ebb3eaf"
format_stamp: "Formatted at 2026-06-19 14:37:48 on dist-test-slave-32hj"
server_key: "fde727016fc37fcbdd3b726a7a1a58cb"
server_key_iv: "c6e9096c72f2941fdb735064ccb1f755"
server_key_version: "encryptionkey@0"
I20260619 14:37:48.777387 6184 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:37:48.779794 6199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:48.781121 6184 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:37:48.781226 6184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/ts-1/wal
uuid: "5fa8307e6d514deeb7b2a7795ebb3eaf"
format_stamp: "Formatted at 2026-06-19 14:37:48 on dist-test-slave-32hj"
server_key: "fde727016fc37fcbdd3b726a7a1a58cb"
server_key_iv: "c6e9096c72f2941fdb735064ccb1f755"
server_key_version: "encryptionkey@0"
I20260619 14:37:48.781340 6184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:37:48.796294 6184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:37:48.799424 6184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:37:48.799595 6184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:37:48.800163 6184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:37:48.800990 6184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:37:48.801044 6184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:48.801100 6184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:37:48.801123 6184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:48.809487 6184 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36829
I20260619 14:37:48.809557 6312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36829 every 8 connection(s)
I20260619 14:37:48.810382 6184 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:37:48.812669 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 6184
I20260619 14:37:48.812810 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:37:48.813073 392 external_mini_cluster.cc:1428] Setting key d7cd0d2b45e955e1f7115840503072e1
Jun 19 14:37:48 dist-test-slave-32hj krb5kdc[5943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879868, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 14:37:48.823678 6027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:48.812149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55003 (local address 127.0.98.62:33905)
0619 14:37:48.812269 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:48.812272 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:48.813124 (+ 852us) server_negotiation.cc:407] Connection header received
0619 14:37:48.814564 (+ 1440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:48.814567 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:48.814619 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:48.814696 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:48.815806 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:48.816331 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:48.817483 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:48.817770 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:48.817862 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:48.820393 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:48.820397 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:48.820399 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:48.820423 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:48.821794 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:48.822258 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:48.822261 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:48.822264 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:48.822341 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:48.822645 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:48.822648 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:48.822650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:48.822826 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:48.822913 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:48.823442 (+ 529us) server_negotiation.cc:299] Negotiation successful
0619 14:37:48.823557 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 14:37:48.824704 6315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:48.812486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33905 (local address 127.0.98.2:55003)
0619 14:37:48.812958 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:48.813009 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:48.814358 (+ 1349us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:48.814845 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:48.814851 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:48.815187 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:48.815670 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:48.815683 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:48.816445 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:48.816447 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:48.817364 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:48.817373 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:48.817910 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:48.817914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:48.818073 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:48.818609 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:48.818626 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:48.820217 (+ 1591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:48.821902 (+ 1685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:48.821909 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:48.821924 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:48.822166 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:48.822425 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:48.822428 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:48.822430 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:48.822564 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:48.822918 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:48.822925 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:48.823170 (+ 245us) client_negotiation.cc:769] Sending connection context
0619 14:37:48.823347 (+ 177us) client_negotiation.cc:240] Negotiation successful
0619 14:37:48.823665 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":106,"threads_started":1}
I20260619 14:37:48.825876 6313 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33905
I20260619 14:37:48.826103 6313 heartbeater.cc:464] Registering TS with master...
I20260619 14:37:48.826567 6313 heartbeater.cc:511] Master 127.0.98.62:33905 requested a full tablet report, sending...
I20260619 14:37:48.827613 5988 ts_manager.cc:194] Registered new tserver with Master: 5fa8307e6d514deeb7b2a7795ebb3eaf (127.0.98.2:36829)
I20260619 14:37:48.828372 5988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55003
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 14:37:48.837761 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33905
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:37:48.944957 6318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:37:48.945231 6318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:37:48.945274 6318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:37:48.949259 6318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:37:48.949311 6318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:37:48.949366 6318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:37:48.954491 6318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:37:48.955452 6318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:37:48.956388 6318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:37:48.965893 6326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:48.965977 6323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:48.965999 6324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:48.966332 6318 server_base.cc:1034] running on GCE node
I20260619 14:37:48.966742 6318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:37:48.967360 6318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:37:48.968530 6318 hybrid_clock.cc:648] HybridClock initialized: now 1781879868968508 us; error 34 us; skew 500 ppm
Jun 19 14:37:48 dist-test-slave-32hj krb5kdc[5943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879868, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:37:48.971246 6318 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:37:48.972103 6318 webserver.cc:460] Webserver started at http://127.0.98.3:42211/ using document root <none> and password file <none>
I20260619 14:37:48.972693 6318 fs_manager.cc:359] Metadata directory not provided
I20260619 14:37:48.972795 6318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:37:48.973029 6318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:37:48.975087 6318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/ts-2/data/instance:
uuid: "babe656cbaca4c8c964bd67ff0d4e8cc"
format_stamp: "Formatted at 2026-06-19 14:37:48 on dist-test-slave-32hj"
server_key: "8df9c3672c97771e94ddefa411f805ac"
server_key_iv: "cb337fe25ee733ccb04ff73cd9c227b4"
server_key_version: "encryptionkey@0"
I20260619 14:37:48.975600 6318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/ts-2/wal/instance:
uuid: "babe656cbaca4c8c964bd67ff0d4e8cc"
format_stamp: "Formatted at 2026-06-19 14:37:48 on dist-test-slave-32hj"
server_key: "8df9c3672c97771e94ddefa411f805ac"
server_key_iv: "cb337fe25ee733ccb04ff73cd9c227b4"
server_key_version: "encryptionkey@0"
I20260619 14:37:48.979127 6318 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:37:48.981506 6333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:48.982791 6318 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:37:48.982903 6318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/ts-2/wal
uuid: "babe656cbaca4c8c964bd67ff0d4e8cc"
format_stamp: "Formatted at 2026-06-19 14:37:48 on dist-test-slave-32hj"
server_key: "8df9c3672c97771e94ddefa411f805ac"
server_key_iv: "cb337fe25ee733ccb04ff73cd9c227b4"
server_key_version: "encryptionkey@0"
I20260619 14:37:48.983027 6318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:37:48.999167 6318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:37:49.002813 6318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:37:49.002992 6318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:37:49.003592 6318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:37:49.004419 6318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:37:49.004470 6318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:49.004513 6318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:37:49.004541 6318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:49.012846 6318 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35229
I20260619 14:37:49.012883 6446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35229 every 8 connection(s)
I20260619 14:37:49.013770 6318 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
Jun 19 14:37:49 dist-test-slave-32hj krb5kdc[5943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879868, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:37:49.023806 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 6318
I20260619 14:37:49.023906 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:37:49.024140 392 external_mini_cluster.cc:1428] Setting key a7d3e94d06bd5d34bef7c58e3bd22f86
I20260619 14:37:49.027480 6027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:49.015548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35139 (local address 127.0.98.62:33905)
0619 14:37:49.015709 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:49.015714 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:49.016583 (+ 869us) server_negotiation.cc:407] Connection header received
0619 14:37:49.017951 (+ 1368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:49.017954 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:49.018003 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:49.018080 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:49.019288 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:49.019823 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:49.020988 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:49.021214 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:49.021318 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:49.023698 (+ 2380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:49.023701 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:49.023704 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:49.023744 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:49.025374 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:49.025858 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:49.025862 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:49.025863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:49.025956 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:49.026258 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:49.026260 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:49.026261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:49.026483 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:49.026560 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:49.027173 (+ 613us) server_negotiation.cc:299] Negotiation successful
0619 14:37:49.027364 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 14:37:49.028236 6449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:49.015966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33905 (local address 127.0.98.3:35139)
0619 14:37:49.016424 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:49.016468 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:49.017729 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:49.018239 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:49.018246 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:49.018619 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:49.019130 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:49.019144 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:49.019931 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:49.019933 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:49.020861 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:49.020878 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:49.021349 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:49.021351 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:49.021484 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:49.022053 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:49.022069 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:49.023505 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:49.025483 (+ 1978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:49.025490 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:49.025508 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:49.025771 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:49.026038 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:49.026040 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:49.026041 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:49.026181 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:49.026610 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:49.026616 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:49.026893 (+ 277us) client_negotiation.cc:769] Sending connection context
0619 14:37:49.027086 (+ 193us) client_negotiation.cc:240] Negotiation successful
0619 14:37:49.027398 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":113,"threads_started":1}
I20260619 14:37:49.029274 6447 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33905
I20260619 14:37:49.029513 6447 heartbeater.cc:464] Registering TS with master...
I20260619 14:37:49.030037 6447 heartbeater.cc:511] Master 127.0.98.62:33905 requested a full tablet report, sending...
I20260619 14:37:49.031092 5988 ts_manager.cc:194] Registered new tserver with Master: babe656cbaca4c8c964bd67ff0d4e8cc (127.0.98.3:35229)
I20260619 14:37:49.031628 5988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35139
I20260619 14:37:49.038403 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/krb5kdc/test-admin.keytab.
Jun 19 14:37:49 dist-test-slave-32hj krb5kdc[5943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:37:49.052965 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260619 14:37:49.055784 6454 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:37:49.054353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42024 (local address 127.0.98.1:43877)
0619 14:37:49.054697 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:49.054710 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:49.054733 (+ 23us) server_negotiation.cc:407] Connection header received
0619 14:37:49.054808 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:49.054813 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:49.054999 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:49.055128 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:49.055597 (+ 469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42024: BlockingRecv error: recv got EOF from 127.0.0.1:42024 (error 108)
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":121,"threads_started":1}
W20260619 14:37:49.055887 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/krb5kdc/krb5cc' of type FILE
Jun 19 14:37:49 dist-test-slave-32hj krb5kdc[5943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:37:49.057025 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 14:37:49 dist-test-slave-32hj krb5kdc[5943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:37:49.058015 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 14:37:49 dist-test-slave-32hj krb5kdc[5943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 14:37:49.067459 6454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:49.059044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42036 (local address 127.0.98.1:43877)
0619 14:37:49.059187 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:49.059191 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:49.059205 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:37:49.059294 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:49.059296 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:49.059333 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:49.059400 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:49.059850 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:49.060625 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:49.061490 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:49.061879 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:49.061963 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:49.064100 (+ 2137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:49.064110 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:49.064123 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:49.064145 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:49.065667 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:49.066081 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:49.066087 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:49.066089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:49.066170 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:49.066480 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:49.066483 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:49.066484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:49.066739 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:49.066943 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:49.067138 (+ 195us) server_negotiation.cc:299] Negotiation successful
0619 14:37:49.067291 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260619 14:37:49.071069 6454 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:37:49.070223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42040 (local address 127.0.98.1:43877)
0619 14:37:49.070346 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:49.070349 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:49.070417 (+ 68us) server_negotiation.cc:407] Connection header received
0619 14:37:49.070505 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:49.070507 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:49.070541 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:49.070599 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:49.070980 (+ 381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42040: BlockingRecv error: recv got EOF from 127.0.0.1:42040 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260619 14:37:49.071141 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/krb5kdc/krb5cc' of type FILE
Jun 19 14:37:49 dist-test-slave-32hj krb5kdc[5943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:37:49.072060 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 14:37:49 dist-test-slave-32hj krb5kdc[5943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:37:49.072971 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 14:37:49 dist-test-slave-32hj krb5kdc[5943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 14:37:49.080585 6454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:49.073731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42056 (local address 127.0.98.1:43877)
0619 14:37:49.073874 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:49.073888 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:49.073903 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:37:49.074021 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:49.074024 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:49.074063 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:49.074126 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:49.074618 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:49.075147 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:49.075898 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:49.076129 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:49.076215 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:49.078247 (+ 2032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:49.078250 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:49.078252 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:49.078273 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:49.079351 (+ 1078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:49.079696 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:49.079698 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:49.079699 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:49.079763 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:49.079957 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:49.079959 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:49.079960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:49.080101 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:49.080189 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:49.080338 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 14:37:49.080428 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260619 14:37:49.082976 6454 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:37:49.082106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42064 (local address 127.0.98.1:43877)
0619 14:37:49.082228 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:49.082231 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:49.082283 (+ 52us) server_negotiation.cc:407] Connection header received
0619 14:37:49.082381 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:49.082383 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:49.082419 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:49.082477 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:49.082879 (+ 402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42064: BlockingRecv error: recv got EOF from 127.0.0.1:42064 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260619 14:37:49.082975 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781879837648036-392-0/krb5kdc/krb5cc' of type FILE
Jun 19 14:37:49 dist-test-slave-32hj krb5kdc[5943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:37:49.083905 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 14:37:49 dist-test-slave-32hj krb5kdc[5943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:37:49.084834 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 14:37:49 dist-test-slave-32hj krb5kdc[5943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 14:37:49.093174 6454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:49.085745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42070 (local address 127.0.98.1:43877)
0619 14:37:49.085861 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:49.085864 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:49.085908 (+ 44us) server_negotiation.cc:407] Connection header received
0619 14:37:49.086005 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:49.086008 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:49.086040 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:49.086100 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:49.086583 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:49.087130 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:49.088057 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:49.088280 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:49.088374 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:49.090547 (+ 2173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:49.090550 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:49.090552 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:49.090580 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:49.091705 (+ 1125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:49.092060 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:49.092062 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:49.092063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:49.092133 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:49.092328 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:49.092330 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:49.092331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:49.092494 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:49.092585 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:49.092900 (+ 315us) server_negotiation.cc:299] Negotiation successful
0619 14:37:49.093016 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 14:37:49.094341 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 6050
I20260619 14:37:49.099071 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 6184
I20260619 14:37:49.103374 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 6318
I20260619 14:37:49.107347 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 5956
2026-06-19T14:37:49Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1980 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:37:49 dist-test-slave-32hj krb5kdc[6467](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:37:49 dist-test-slave-32hj krb5kdc[6467](info): set up 2 sockets
Jun 19 14:37:49 dist-test-slave-32hj krb5kdc[6467](info): commencing operation
krb5kdc: starting...
W20260619 14:37:50.072959 392 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/krb5kdc/test-user.keytab.
Jun 19 14:37:50 dist-test-slave-32hj krb5kdc[6467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:37:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:37:50Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260619 14:37:50.191509 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46665
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46665
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:37:50.296763 6480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:37:50.297035 6480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:37:50.297075 6480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:37:50.301010 6480 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:37:50.301065 6480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:37:50.301082 6480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:37:50.301098 6480 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:37:50.301112 6480 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:37:50.306285 6480 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46665
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:37:50.307310 6480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:37:50.308264 6480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:37:50.316565 6485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:50.316636 6480 server_base.cc:1034] running on GCE node
W20260619 14:37:50.316545 6486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:50.316551 6488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:50.317457 6480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:37:50.318538 6480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:37:50.319687 6480 hybrid_clock.cc:648] HybridClock initialized: now 1781879870319666 us; error 35 us; skew 500 ppm
Jun 19 14:37:50 dist-test-slave-32hj krb5kdc[6467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879870, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:37:50.325178 6480 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260619 14:37:50.326083 6480 webserver.cc:460] Webserver started at http://127.0.98.62:39475/ using document root <none> and password file <none>
I20260619 14:37:50.326606 6480 fs_manager.cc:359] Metadata directory not provided
I20260619 14:37:50.326663 6480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:37:50.326876 6480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:37:50.328979 6480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-0/data/instance:
uuid: "e41b19ce379e45198aeff6fd3814bb43"
format_stamp: "Formatted at 2026-06-19 14:37:50 on dist-test-slave-32hj"
server_key: "69a36659f325340824bdbe0475d256c8"
server_key_iv: "ba92fe8e7f0e5cb24ce007f327ed7981"
server_key_version: "encryptionkey@0"
I20260619 14:37:50.329485 6480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-0/wal/instance:
uuid: "e41b19ce379e45198aeff6fd3814bb43"
format_stamp: "Formatted at 2026-06-19 14:37:50 on dist-test-slave-32hj"
server_key: "69a36659f325340824bdbe0475d256c8"
server_key_iv: "ba92fe8e7f0e5cb24ce007f327ed7981"
server_key_version: "encryptionkey@0"
I20260619 14:37:50.333096 6480 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:37:50.335427 6495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:50.336863 6480 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:37:50.336982 6480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-0/wal
uuid: "e41b19ce379e45198aeff6fd3814bb43"
format_stamp: "Formatted at 2026-06-19 14:37:50 on dist-test-slave-32hj"
server_key: "69a36659f325340824bdbe0475d256c8"
server_key_iv: "ba92fe8e7f0e5cb24ce007f327ed7981"
server_key_version: "encryptionkey@0"
I20260619 14:37:50.337092 6480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:37:50.346756 6480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:37:50.349339 6480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:37:50.349489 6480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:37:50.355826 6480 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46665
I20260619 14:37:50.356086 6547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46665 every 8 connection(s)
I20260619 14:37:50.356698 6480 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:37:50.357465 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 6480
I20260619 14:37:50.357600 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-0/wal/instance
I20260619 14:37:50.357905 392 external_mini_cluster.cc:1428] Setting key 43894c73d90f1e220e97942e5ff87ce2
I20260619 14:37:50.359867 6548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:50.366083 6548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e41b19ce379e45198aeff6fd3814bb43: Bootstrap starting.
Jun 19 14:37:50 dist-test-slave-32hj krb5kdc[6467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 14:37:50.368567 6548 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e41b19ce379e45198aeff6fd3814bb43: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:50.369405 6548 log.cc:824] T 00000000000000000000000000000000 P e41b19ce379e45198aeff6fd3814bb43: Log is configured to *not* fsync() on all Append() calls
I20260619 14:37:50.371173 6548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e41b19ce379e45198aeff6fd3814bb43: No bootstrap required, opened a new log
I20260619 14:37:50.371773 6551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:50.359139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48312 (local address 127.0.98.62:46665)
0619 14:37:50.359531 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:50.359548 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:50.359589 (+ 41us) server_negotiation.cc:407] Connection header received
0619 14:37:50.360393 (+ 804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:50.360415 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:50.360829 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:50.361173 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:50.361782 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:50.363269 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:50.364059 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:50.364712 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:50.364898 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:50.367191 (+ 2293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:50.367197 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:50.367213 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:50.367245 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:50.369133 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:50.369565 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:50.369572 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:50.369576 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:50.369685 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:50.369961 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:50.369964 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:50.369966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:50.370359 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:50.370524 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:50.370800 (+ 276us) server_negotiation.cc:299] Negotiation successful
0619 14:37:50.371061 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":124,"threads_started":1}
I20260619 14:37:50.374830 6548 raft_consensus.cc:348] T 00000000000000000000000000000000 P e41b19ce379e45198aeff6fd3814bb43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e41b19ce379e45198aeff6fd3814bb43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46665 } }
I20260619 14:37:50.375046 6548 raft_consensus.cc:374] T 00000000000000000000000000000000 P e41b19ce379e45198aeff6fd3814bb43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:50.375099 6548 raft_consensus.cc:729] T 00000000000000000000000000000000 P e41b19ce379e45198aeff6fd3814bb43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e41b19ce379e45198aeff6fd3814bb43, State: Initialized, Role: FOLLOWER
I20260619 14:37:50.375488 6548 consensus_queue.cc:260] T 00000000000000000000000000000000 P e41b19ce379e45198aeff6fd3814bb43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e41b19ce379e45198aeff6fd3814bb43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46665 } }
I20260619 14:37:50.375612 6548 raft_consensus.cc:388] T 00000000000000000000000000000000 P e41b19ce379e45198aeff6fd3814bb43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:37:50.375672 6548 raft_consensus.cc:482] T 00000000000000000000000000000000 P e41b19ce379e45198aeff6fd3814bb43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:37:50.375763 6548 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e41b19ce379e45198aeff6fd3814bb43 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:50.376567 6548 raft_consensus.cc:504] T 00000000000000000000000000000000 P e41b19ce379e45198aeff6fd3814bb43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e41b19ce379e45198aeff6fd3814bb43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46665 } }
I20260619 14:37:50.376905 6548 leader_election.cc:302] T 00000000000000000000000000000000 P e41b19ce379e45198aeff6fd3814bb43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e41b19ce379e45198aeff6fd3814bb43; no voters:
I20260619 14:37:50.377169 6548 leader_election.cc:288] T 00000000000000000000000000000000 P e41b19ce379e45198aeff6fd3814bb43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:37:50.377305 6553 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e41b19ce379e45198aeff6fd3814bb43 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:37:50.377498 6553 raft_consensus.cc:686] T 00000000000000000000000000000000 P e41b19ce379e45198aeff6fd3814bb43 [term 1 LEADER]: Becoming Leader. State: Replica: e41b19ce379e45198aeff6fd3814bb43, State: Running, Role: LEADER
I20260619 14:37:50.377815 6553 consensus_queue.cc:237] T 00000000000000000000000000000000 P e41b19ce379e45198aeff6fd3814bb43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e41b19ce379e45198aeff6fd3814bb43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46665 } }
I20260619 14:37:50.378324 6548 sys_catalog.cc:564] T 00000000000000000000000000000000 P e41b19ce379e45198aeff6fd3814bb43 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:37:50.379269 6554 sys_catalog.cc:455] T 00000000000000000000000000000000 P e41b19ce379e45198aeff6fd3814bb43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e41b19ce379e45198aeff6fd3814bb43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e41b19ce379e45198aeff6fd3814bb43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46665 } } }
I20260619 14:37:50.379395 6554 sys_catalog.cc:458] T 00000000000000000000000000000000 P e41b19ce379e45198aeff6fd3814bb43 [sys.catalog]: This master's current role is: LEADER
I20260619 14:37:50.379484 6555 sys_catalog.cc:455] T 00000000000000000000000000000000 P e41b19ce379e45198aeff6fd3814bb43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e41b19ce379e45198aeff6fd3814bb43. Latest consensus state: current_term: 1 leader_uuid: "e41b19ce379e45198aeff6fd3814bb43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e41b19ce379e45198aeff6fd3814bb43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46665 } } }
I20260619 14:37:50.379566 6555 sys_catalog.cc:458] T 00000000000000000000000000000000 P e41b19ce379e45198aeff6fd3814bb43 [sys.catalog]: This master's current role is: LEADER
I20260619 14:37:50.379969 6562 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:37:50.382529 6562 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:37:50.387851 6562 catalog_manager.cc:1347] Generated new cluster ID: a546ae03a7f4400d99fa082375d31b43
I20260619 14:37:50.387916 6562 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:37:50.417394 6562 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:37:50.418318 6562 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:37:50.422461 6562 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e41b19ce379e45198aeff6fd3814bb43: Generated new TSK 0
I20260619 14:37:50.423111 6562 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260619 14:37:50.452350 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46665
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:37:50.559825 6574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:37:50.560096 6574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:37:50.560154 6574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:37:50.564373 6574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:37:50.564442 6574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:37:50.564529 6574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:37:50.570003 6574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46665
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:37:50.571063 6574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:37:50.572083 6574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:37:50.581411 6579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:50.581566 6574 server_base.cc:1034] running on GCE node
W20260619 14:37:50.581440 6582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:50.581429 6580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:50.582072 6574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:37:50.582703 6574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:37:50.583877 6574 hybrid_clock.cc:648] HybridClock initialized: now 1781879870583857 us; error 30 us; skew 500 ppm
Jun 19 14:37:50 dist-test-slave-32hj krb5kdc[6467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879870, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:37:50.586591 6574 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260619 14:37:50.587428 6574 webserver.cc:460] Webserver started at http://127.0.98.1:33719/ using document root <none> and password file <none>
I20260619 14:37:50.587981 6574 fs_manager.cc:359] Metadata directory not provided
I20260619 14:37:50.588045 6574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:37:50.588224 6574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:37:50.590217 6574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-0/data/instance:
uuid: "57ffea6d72ae469491ce019f5e03a885"
format_stamp: "Formatted at 2026-06-19 14:37:50 on dist-test-slave-32hj"
server_key: "ba2e86ddefe9b0fb79ece617497ff03a"
server_key_iv: "a60aa1529aa6f9703f002ccae9c7ab56"
server_key_version: "encryptionkey@0"
I20260619 14:37:50.590730 6574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-0/wal/instance:
uuid: "57ffea6d72ae469491ce019f5e03a885"
format_stamp: "Formatted at 2026-06-19 14:37:50 on dist-test-slave-32hj"
server_key: "ba2e86ddefe9b0fb79ece617497ff03a"
server_key_iv: "a60aa1529aa6f9703f002ccae9c7ab56"
server_key_version: "encryptionkey@0"
I20260619 14:37:50.594259 6574 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260619 14:37:50.596540 6589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:50.597784 6574 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:37:50.597888 6574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-0/wal
uuid: "57ffea6d72ae469491ce019f5e03a885"
format_stamp: "Formatted at 2026-06-19 14:37:50 on dist-test-slave-32hj"
server_key: "ba2e86ddefe9b0fb79ece617497ff03a"
server_key_iv: "a60aa1529aa6f9703f002ccae9c7ab56"
server_key_version: "encryptionkey@0"
I20260619 14:37:50.597985 6574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:37:50.617031 6574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:37:50.620368 6574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:37:50.620558 6574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:37:50.621169 6574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:37:50.621995 6574 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:37:50.622042 6574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:50.622094 6574 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:37:50.622121 6574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:50.630508 6574 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33487
I20260619 14:37:50.630591 6702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33487 every 8 connection(s)
I20260619 14:37:50.631417 6574 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:37:50.638737 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 6574
I20260619 14:37:50.638842 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:37:50.639137 392 external_mini_cluster.cc:1428] Setting key 9004acf7c5c39ad153c6cc3d6355da10
Jun 19 14:37:50 dist-test-slave-32hj krb5kdc[6467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879870, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 14:37:50.644825 6551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:50.633252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44739 (local address 127.0.98.62:46665)
0619 14:37:50.633431 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:50.633436 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:50.634299 (+ 863us) server_negotiation.cc:407] Connection header received
0619 14:37:50.635581 (+ 1282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:50.635585 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:50.635629 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:50.635700 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:50.636844 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:50.637373 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:50.638546 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:50.638776 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:50.638888 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:50.641553 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:50.641556 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:50.641558 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:50.641580 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:50.642898 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:50.643360 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:50.643364 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:50.643365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:50.643441 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:50.643739 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:50.643742 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:50.643744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:50.643939 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:50.644024 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:50.644520 (+ 496us) server_negotiation.cc:299] Negotiation successful
0619 14:37:50.644641 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 14:37:50.645655 6705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:50.633678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46665 (local address 127.0.98.1:44739)
0619 14:37:50.634132 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:50.634188 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:50.635400 (+ 1212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:50.635856 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:50.635863 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:50.636207 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:50.636682 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:50.636695 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:50.637496 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:50.637499 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:50.638428 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:50.638438 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:50.638931 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:50.638935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:50.639136 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:50.639786 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:50.639803 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:50.641403 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:50.643004 (+ 1601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:50.643010 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:50.643026 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:50.643271 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:50.643519 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:50.643522 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:50.643524 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:50.643658 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:50.644037 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:50.644044 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:50.644290 (+ 246us) client_negotiation.cc:769] Sending connection context
0619 14:37:50.644466 (+ 176us) client_negotiation.cc:240] Negotiation successful
0619 14:37:50.644823 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":117,"threads_started":1}
I20260619 14:37:50.646938 6703 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46665
I20260619 14:37:50.647161 6703 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 14:37:50.647622 6703 heartbeater.cc:511] Master 127.0.98.62:46665 requested a full tablet report, sending...
I20260619 14:37:50.649142 6512 ts_manager.cc:194] Registered new tserver with Master: 57ffea6d72ae469491ce019f5e03a885 (127.0.98.1:33487)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260619 14:37:50.650460 6512 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44739
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260619 14:37:50.665758 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46665
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:37:50.772989 6708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:37:50.773253 6708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:37:50.773294 6708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:37:50.777287 6708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:37:50.777348 6708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:37:50.777411 6708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:37:50.782716 6708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46665
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:37:50.783737 6708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:37:50.784741 6708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:37:50.793844 6714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:50.793929 6708 server_base.cc:1034] running on GCE node
W20260619 14:37:50.793879 6713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:50.794095 6716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:50.794456 6708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:37:50.795054 6708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:37:50.796209 6708 hybrid_clock.cc:648] HybridClock initialized: now 1781879870796187 us; error 35 us; skew 500 ppm
Jun 19 14:37:50 dist-test-slave-32hj krb5kdc[6467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879870, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:37:50.798920 6708 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260619 14:37:50.799739 6708 webserver.cc:460] Webserver started at http://127.0.98.2:46665/ using document root <none> and password file <none>
I20260619 14:37:50.800261 6708 fs_manager.cc:359] Metadata directory not provided
I20260619 14:37:50.800325 6708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:37:50.800544 6708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:37:50.802664 6708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-1/data/instance:
uuid: "24dc9c1322cb4b5092de8d6a4c742d77"
format_stamp: "Formatted at 2026-06-19 14:37:50 on dist-test-slave-32hj"
server_key: "5ad151613ecbcf214f60f1f122e25a77"
server_key_iv: "fef86f629144ef2ab94a2bcfac6ad2b7"
server_key_version: "encryptionkey@0"
I20260619 14:37:50.803185 6708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-1/wal/instance:
uuid: "24dc9c1322cb4b5092de8d6a4c742d77"
format_stamp: "Formatted at 2026-06-19 14:37:50 on dist-test-slave-32hj"
server_key: "5ad151613ecbcf214f60f1f122e25a77"
server_key_iv: "fef86f629144ef2ab94a2bcfac6ad2b7"
server_key_version: "encryptionkey@0"
I20260619 14:37:50.806830 6708 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:37:50.809273 6723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:50.810557 6708 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:37:50.810672 6708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-1/wal
uuid: "24dc9c1322cb4b5092de8d6a4c742d77"
format_stamp: "Formatted at 2026-06-19 14:37:50 on dist-test-slave-32hj"
server_key: "5ad151613ecbcf214f60f1f122e25a77"
server_key_iv: "fef86f629144ef2ab94a2bcfac6ad2b7"
server_key_version: "encryptionkey@0"
I20260619 14:37:50.810776 6708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:37:50.828583 6708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:37:50.831696 6708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:37:50.831879 6708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:37:50.832465 6708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:37:50.833309 6708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:37:50.833356 6708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:50.833408 6708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:37:50.833437 6708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:50.841406 6708 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46221
I20260619 14:37:50.841460 6836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46221 every 8 connection(s)
I20260619 14:37:50.842281 6708 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 14:37:50 dist-test-slave-32hj krb5kdc[6467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879870, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 14:37:50.851672 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 6708
I20260619 14:37:50.851794 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:37:50.852053 392 external_mini_cluster.cc:1428] Setting key 70fb7b4b14e1e50b654adbdb08c8705d
I20260619 14:37:50.855687 6551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:50.844001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45625 (local address 127.0.98.62:46665)
0619 14:37:50.844121 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:50.844124 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:50.844990 (+ 866us) server_negotiation.cc:407] Connection header received
0619 14:37:50.846302 (+ 1312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:50.846306 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:50.846365 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:50.846439 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:50.847638 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:50.848156 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:50.849341 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:50.849561 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:50.849649 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:50.852147 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:50.852151 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:50.852154 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:50.852182 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:50.853724 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:50.854212 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:50.854216 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:50.854218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:50.854316 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:50.854605 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:50.854607 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:50.854609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:50.854803 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:50.854893 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:50.855400 (+ 507us) server_negotiation.cc:299] Negotiation successful
0619 14:37:50.855532 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 14:37:50.856554 6839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:50.844386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46665 (local address 127.0.98.2:45625)
0619 14:37:50.844851 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:50.844895 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:50.846089 (+ 1194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:50.846627 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:50.846635 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:50.846997 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:50.847491 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:50.847505 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:50.848275 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:50.848278 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:50.849225 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:50.849235 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:50.849702 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:50.849706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:50.849864 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:50.850424 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:50.850441 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:50.851962 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:50.853849 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:50.853854 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:50.853866 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:50.854121 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:50.854398 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:50.854400 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:50.854401 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:50.854533 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:50.854904 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:50.854909 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:50.855164 (+ 255us) client_negotiation.cc:769] Sending connection context
0619 14:37:50.855334 (+ 170us) client_negotiation.cc:240] Negotiation successful
0619 14:37:50.855659 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":112,"threads_started":1}
I20260619 14:37:50.857775 6837 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46665
I20260619 14:37:50.858016 6837 heartbeater.cc:464] Registering TS with master...
I20260619 14:37:50.858470 6837 heartbeater.cc:511] Master 127.0.98.62:46665 requested a full tablet report, sending...
I20260619 14:37:50.859550 6512 ts_manager.cc:194] Registered new tserver with Master: 24dc9c1322cb4b5092de8d6a4c742d77 (127.0.98.2:46221)
I20260619 14:37:50.860103 6512 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45625
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260619 14:37:50.876865 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46665
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:37:50.983209 6842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:37:50.983466 6842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:37:50.983508 6842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:37:50.987437 6842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:37:50.987499 6842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:37:50.987557 6842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:37:50.993023 6842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46665
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:37:50.994249 6842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:37:50.995435 6842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:37:51.005405 6848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:51.005394 6850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:51.005427 6842 server_base.cc:1034] running on GCE node
W20260619 14:37:51.005394 6847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:51.005991 6842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:37:51.006534 6842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:37:51.007691 6842 hybrid_clock.cc:648] HybridClock initialized: now 1781879871007667 us; error 33 us; skew 500 ppm
Jun 19 14:37:51 dist-test-slave-32hj krb5kdc[6467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879871, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:37:51.012988 6842 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260619 14:37:51.013833 6842 webserver.cc:460] Webserver started at http://127.0.98.3:40301/ using document root <none> and password file <none>
I20260619 14:37:51.014374 6842 fs_manager.cc:359] Metadata directory not provided
I20260619 14:37:51.014436 6842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:37:51.014647 6842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:37:51.016628 6842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-2/data/instance:
uuid: "77032b2db22744feb997087f373b1209"
format_stamp: "Formatted at 2026-06-19 14:37:51 on dist-test-slave-32hj"
server_key: "9c09b8adc44e52152472901bd7c47ede"
server_key_iv: "df202399ce48ccc156a33c5030bb347f"
server_key_version: "encryptionkey@0"
I20260619 14:37:51.017180 6842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-2/wal/instance:
uuid: "77032b2db22744feb997087f373b1209"
format_stamp: "Formatted at 2026-06-19 14:37:51 on dist-test-slave-32hj"
server_key: "9c09b8adc44e52152472901bd7c47ede"
server_key_iv: "df202399ce48ccc156a33c5030bb347f"
server_key_version: "encryptionkey@0"
I20260619 14:37:51.020645 6842 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:37:51.022921 6857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:51.024046 6842 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:37:51.024137 6842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-2/wal
uuid: "77032b2db22744feb997087f373b1209"
format_stamp: "Formatted at 2026-06-19 14:37:51 on dist-test-slave-32hj"
server_key: "9c09b8adc44e52152472901bd7c47ede"
server_key_iv: "df202399ce48ccc156a33c5030bb347f"
server_key_version: "encryptionkey@0"
I20260619 14:37:51.024238 6842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:37:51.046036 6842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:37:51.048852 6842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:37:51.049017 6842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:37:51.049569 6842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:37:51.050338 6842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:37:51.050381 6842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:51.050421 6842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:37:51.050449 6842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:51.058512 6842 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36403
I20260619 14:37:51.058547 6970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36403 every 8 connection(s)
I20260619 14:37:51.059347 6842 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:37:51.062783 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 6842
I20260619 14:37:51.062888 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:37:51.063155 392 external_mini_cluster.cc:1428] Setting key b6239287ee64783f0e58ba31fdee54f4
Jun 19 14:37:51 dist-test-slave-32hj krb5kdc[6467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879871, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 14:37:51.071920 6551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:51.061020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56417 (local address 127.0.98.62:46665)
0619 14:37:51.061166 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:51.061171 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:51.062022 (+ 851us) server_negotiation.cc:407] Connection header received
0619 14:37:51.063342 (+ 1320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:51.063346 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:51.063399 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:51.063493 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:51.064858 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.065435 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.066521 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.066724 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.066792 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:51.069030 (+ 2238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:51.069034 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:51.069036 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:51.069059 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:51.070355 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:51.070747 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:51.070749 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:51.070750 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:51.070814 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:51.071073 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:51.071075 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:51.071076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:51.071234 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:51.071302 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:51.071696 (+ 394us) server_negotiation.cc:299] Negotiation successful
0619 14:37:51.071784 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 14:37:51.072772 6973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:51.061426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46665 (local address 127.0.98.3:56417)
0619 14:37:51.061879 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:51.061921 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:51.063136 (+ 1215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:51.063689 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:51.063698 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:51.064130 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:51.064678 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:51.064692 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.065539 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.065542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:51.066431 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:51.066440 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.066843 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.066846 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:51.066983 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:51.067478 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:51.067493 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:51.068889 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:51.070435 (+ 1546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:51.070439 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:51.070450 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:51.070679 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:51.070893 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:51.070895 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:51.070897 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:51.071012 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:51.071301 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:51.071306 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:51.071523 (+ 217us) client_negotiation.cc:769] Sending connection context
0619 14:37:51.071669 (+ 146us) client_negotiation.cc:240] Negotiation successful
0619 14:37:51.071925 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":109,"threads_started":1}
I20260619 14:37:51.073841 6971 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46665
I20260619 14:37:51.074069 6971 heartbeater.cc:464] Registering TS with master...
I20260619 14:37:51.074529 6971 heartbeater.cc:511] Master 127.0.98.62:46665 requested a full tablet report, sending...
I20260619 14:37:51.075512 6512 ts_manager.cc:194] Registered new tserver with Master: 77032b2db22744feb997087f373b1209 (127.0.98.3:36403)
I20260619 14:37:51.076170 6512 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56417
I20260619 14:37:51.076859 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260619 14:37:51.101562 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39319
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46665,127.0.98.61:39319
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:37:51.219863 6976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:37:51.220117 6976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:37:51.220168 6976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:37:51.224175 6976 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:37:51.224242 6976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:37:51.224260 6976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:37:51.224277 6976 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:37:51.224290 6976 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:37:51.229607 6976 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46665,127.0.98.61:39319
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
W20260619 14:37:51.229792 6976 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:46665,127.0.98.61:39319'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260619 14:37:51.230635 6976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:37:51.231613 6976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:37:51.239802 6982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:51.239796 6981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:51.239971 6976 server_base.cc:1034] running on GCE node
W20260619 14:37:51.240089 6984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:51.240690 6976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:37:51.241835 6976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:37:51.243016 6976 hybrid_clock.cc:648] HybridClock initialized: now 1781879871242987 us; error 40 us; skew 500 ppm
Jun 19 14:37:51 dist-test-slave-32hj krb5kdc[6467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879871, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:37:51.245677 6976 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260619 14:37:51.246522 6976 webserver.cc:460] Webserver started at http://127.0.98.61:35509/ using document root <none> and password file <none>
I20260619 14:37:51.247040 6976 fs_manager.cc:359] Metadata directory not provided
I20260619 14:37:51.247103 6976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:37:51.247284 6976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:37:51.249353 6976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-1/data/instance:
uuid: "6731bc26510d493c99ecc1aba97fd05b"
format_stamp: "Formatted at 2026-06-19 14:37:51 on dist-test-slave-32hj"
server_key: "592f95d9f9a2d16e26653fc44bb8bdac"
server_key_iv: "c935ef08b03e421e200492e18742fd06"
server_key_version: "encryptionkey@0"
I20260619 14:37:51.249881 6976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-1/wal/instance:
uuid: "6731bc26510d493c99ecc1aba97fd05b"
format_stamp: "Formatted at 2026-06-19 14:37:51 on dist-test-slave-32hj"
server_key: "592f95d9f9a2d16e26653fc44bb8bdac"
server_key_iv: "c935ef08b03e421e200492e18742fd06"
server_key_version: "encryptionkey@0"
I20260619 14:37:51.253585 6976 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:37:51.255897 6991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:51.257102 6976 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:37:51.257211 6976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-1/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-1/wal
uuid: "6731bc26510d493c99ecc1aba97fd05b"
format_stamp: "Formatted at 2026-06-19 14:37:51 on dist-test-slave-32hj"
server_key: "592f95d9f9a2d16e26653fc44bb8bdac"
server_key_iv: "c935ef08b03e421e200492e18742fd06"
server_key_version: "encryptionkey@0"
I20260619 14:37:51.257304 6976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:37:51.270789 6976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:37:51.273720 6976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:37:51.273891 6976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:37:51.280548 6976 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:39319
I20260619 14:37:51.280838 7043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:39319 every 8 connection(s)
I20260619 14:37:51.281530 6976 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-1/data/info.pb
I20260619 14:37:51.284143 7044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:51.287437 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 6976
I20260619 14:37:51.287760 7044 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46665 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:37:51 dist-test-slave-32hj krb5kdc[6467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879871, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 19 14:37:51 dist-test-slave-32hj krb5kdc[6467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 14:37:51.299386 6551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:51.289771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48328 (local address 127.0.98.62:46665)
0619 14:37:51.289919 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:51.289924 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:51.290129 (+ 205us) server_negotiation.cc:407] Connection header received
0619 14:37:51.290646 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:51.290649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:51.290701 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:51.290778 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:51.291603 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.292149 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.293199 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.293414 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.293501 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:51.295940 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:51.295945 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:51.295948 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:51.295976 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:51.297404 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:51.297941 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:51.297944 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:51.297946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:51.298018 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:51.298328 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:51.298331 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:51.298333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:51.298556 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:51.298638 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:51.299102 (+ 464us) server_negotiation.cc:299] Negotiation successful
0619 14:37:51.299216 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 14:37:51.300199 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:51.288969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47396 (local address 127.0.98.61:39319)
0619 14:37:51.289465 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:51.289483 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:51.289521 (+ 38us) server_negotiation.cc:407] Connection header received
0619 14:37:51.290388 (+ 867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:51.290406 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:51.290599 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:51.290766 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:51.291563 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.292403 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.293227 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.293588 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.293694 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:51.296310 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:51.296317 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:51.296330 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:51.296354 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:51.297907 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:51.298294 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:51.298300 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:51.298301 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:51.298369 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:51.298607 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:51.298610 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:51.298612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:51.298895 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:51.299068 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:51.299270 (+ 202us) server_negotiation.cc:299] Negotiation successful
0619 14:37:51.299555 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":348,"spinlock_wait_cycles":9088,"thread_start_us":137,"threads_started":1}
I20260619 14:37:51.300197 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:51.289646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46665 (local address 127.0.0.1:48328)
0619 14:37:51.290000 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:51.290040 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:51.290418 (+ 378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:51.290850 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:51.290855 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:51.291201 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:51.291459 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:51.291475 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.292280 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.292283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:51.293086 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:51.293095 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.293544 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.293547 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:51.293690 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:51.294255 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:51.294276 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:51.295778 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:51.297512 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:51.297520 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:51.297529 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:51.297853 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:51.298094 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:51.298097 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:51.298100 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:51.298229 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:51.298650 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:51.298657 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:51.298902 (+ 245us) client_negotiation.cc:769] Sending connection context
0619 14:37:51.299045 (+ 143us) client_negotiation.cc:240] Negotiation successful
0619 14:37:51.299374 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":94,"threads_started":1}
I20260619 14:37:51.301545 7044 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39319 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:37:51 dist-test-slave-32hj krb5kdc[6467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879871, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 14:37:51.308604 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:51.302048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39319 (local address 127.0.0.1:47410)
0619 14:37:51.302134 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:51.302147 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:51.302216 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:51.302414 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:51.302417 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:51.302554 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:51.302638 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:51.302643 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.303309 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.303312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:51.303823 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:51.303829 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.304219 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.304222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:51.304331 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:51.304814 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:51.304825 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:51.305925 (+ 1100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:51.307364 (+ 1439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:51.307367 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:51.307369 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:51.307533 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:51.307752 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:51.307755 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:51.307757 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:51.307842 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:51.308150 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:51.308152 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:51.308228 (+ 76us) client_negotiation.cc:769] Sending connection context
0619 14:37:51.308307 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 14:37:51.308432 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":2}
I20260619 14:37:51.308604 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:51.302160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47410 (local address 127.0.98.61:39319)
0619 14:37:51.302263 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:51.302266 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:51.302277 (+ 11us) server_negotiation.cc:407] Connection header received
0619 14:37:51.302303 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:51.302305 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:51.302338 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:51.302399 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:51.302740 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.303226 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.303914 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.304136 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.304222 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:51.306038 (+ 1816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:51.306042 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:51.306045 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:51.306066 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:51.307255 (+ 1189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:51.307611 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:51.307614 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:51.307616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:51.307680 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:51.307914 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:51.307917 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:51.307919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:51.308074 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:51.308150 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:51.308362 (+ 212us) server_negotiation.cc:299] Negotiation successful
0619 14:37:51.308476 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":51}
I20260619 14:37:51.311542 7044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b: Bootstrap starting.
I20260619 14:37:51.313921 7044 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:51.314797 7044 log.cc:824] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b: Log is configured to *not* fsync() on all Append() calls
I20260619 14:37:51.316877 7044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b: No bootstrap required, opened a new log
I20260619 14:37:51.319809 7044 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e41b19ce379e45198aeff6fd3814bb43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46665 } } peers { permanent_uuid: "6731bc26510d493c99ecc1aba97fd05b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39319 } }
I20260619 14:37:51.320003 7044 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:51.320036 7044 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6731bc26510d493c99ecc1aba97fd05b, State: Initialized, Role: FOLLOWER
I20260619 14:37:51.320466 7044 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e41b19ce379e45198aeff6fd3814bb43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46665 } } peers { permanent_uuid: "6731bc26510d493c99ecc1aba97fd05b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39319 } }
I20260619 14:37:51.320967 7050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e41b19ce379e45198aeff6fd3814bb43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46665 } } peers { permanent_uuid: "6731bc26510d493c99ecc1aba97fd05b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39319 } } }
I20260619 14:37:51.321081 7050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:37:51.321686 7044 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b [sys.catalog]: configured and running, proceeding with master startup.
W20260619 14:37:51.326081 7061 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:37:51.326164 7061 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:37:51.328168 6976 master_runner.cc:428] Detected that this master 6731bc26510d493c99ecc1aba97fd05b is joining an existing cluster
I20260619 14:37:51.328224 6976 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:39319
I20260619 14:37:51.337757 7067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:51.330206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46665 (local address 127.0.0.1:48330)
0619 14:37:51.330470 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:51.330485 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:51.330585 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:51.330959 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:51.330962 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:51.331185 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:51.331329 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:51.331336 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.332369 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.332372 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:51.333246 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:51.333253 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.333658 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.333662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:51.333743 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:51.334212 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:51.334226 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:51.334663 (+ 437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:51.336229 (+ 1566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:51.336232 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:51.336234 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:51.336474 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:51.336705 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:51.336708 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:51.336710 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:51.336840 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:51.337243 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:51.337245 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:51.337342 (+ 97us) client_negotiation.cc:769] Sending connection context
0619 14:37:51.337443 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 14:37:51.337568 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":89,"threads_started":1}
I20260619 14:37:51.338002 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:51.330672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47422 (local address 127.0.98.61:39319)
0619 14:37:51.330776 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:51.330780 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:51.331039 (+ 259us) server_negotiation.cc:407] Connection header received
0619 14:37:51.331121 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:51.331124 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:51.331168 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:51.331255 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:51.331655 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.332120 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.333247 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.333582 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.333679 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:51.334978 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:51.334982 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:51.334984 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:51.335005 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:51.336508 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:51.336983 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:51.336986 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:51.336988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:51.337054 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:51.337293 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:51.337296 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:51.337297 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:51.337465 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:51.337539 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:51.337778 (+ 239us) server_negotiation.cc:299] Negotiation successful
0619 14:37:51.337884 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 14:37:51.338274 7068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:51.330664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39319 (local address 127.0.0.1:47422)
0619 14:37:51.330954 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:51.330965 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:51.331045 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:51.331277 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:51.331280 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:51.331484 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:51.331567 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:51.331572 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.332219 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.332222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:51.333128 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:51.333137 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.333795 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.333798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:51.333885 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:51.334365 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:51.334378 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:51.334854 (+ 476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:51.336609 (+ 1755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:51.336612 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:51.336614 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:51.336890 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:51.337138 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:51.337141 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:51.337143 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:51.337211 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:51.337570 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:51.337572 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:51.337641 (+ 69us) client_negotiation.cc:769] Sending connection context
0619 14:37:51.338091 (+ 450us) client_negotiation.cc:240] Negotiation successful
0619 14:37:51.338175 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":66,"threads_started":1}
I20260619 14:37:51.337791 6551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:51.330319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48330 (local address 127.0.98.62:46665)
0619 14:37:51.330445 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:51.330450 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:51.330556 (+ 106us) server_negotiation.cc:407] Connection header received
0619 14:37:51.330704 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:51.330708 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:51.330762 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:51.330815 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:51.331444 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.332263 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.333337 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.333559 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.333643 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:51.334783 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:51.334787 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:51.334789 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:51.334821 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:51.336138 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:51.336552 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:51.336556 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:51.336558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:51.336632 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:51.336956 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:51.336960 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:51.336962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:51.337134 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:51.337190 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:51.337515 (+ 325us) server_negotiation.cc:299] Negotiation successful
0619 14:37:51.337628 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Jun 19 14:37:51 dist-test-slave-32hj krb5kdc[6467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879870, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 14:37:51.350840 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:51.343063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47426 (local address 127.0.98.61:39319)
0619 14:37:51.343163 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:51.343167 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:51.343474 (+ 307us) server_negotiation.cc:407] Connection header received
0619 14:37:51.343643 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:51.343646 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:51.343695 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:51.343756 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:51.344369 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.344888 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.345696 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.345910 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.345988 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:51.348141 (+ 2153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:51.348144 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:51.348146 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:51.348165 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:51.349363 (+ 1198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:51.349760 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:51.349764 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:51.349766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:51.349829 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:51.350051 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:51.350054 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:51.350056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:51.350212 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:51.350288 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:51.350587 (+ 299us) server_negotiation.cc:299] Negotiation successful
0619 14:37:51.350686 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 14:37:51.350853 7069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:51.343027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39319 (local address 127.0.0.1:47426)
0619 14:37:51.343363 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:51.343405 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:51.343529 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:51.343802 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:51.343808 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:51.344154 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:51.344270 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:51.344276 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.344976 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.344980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:51.345589 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:51.345597 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.345997 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.346001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:51.346097 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:51.346544 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:51.346556 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:51.348003 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:51.349454 (+ 1451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:51.349462 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:51.349469 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:51.349678 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:51.349902 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:51.349905 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:51.349907 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:51.349980 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:51.350302 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:51.350309 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:51.350414 (+ 105us) client_negotiation.cc:769] Sending connection context
0619 14:37:51.350530 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 14:37:51.350707 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":118,"threads_started":1}
I20260619 14:37:51.352104 6512 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:39319: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "6731bc26510d493c99ecc1aba97fd05b"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 39319
}
attrs {
promote: true
}
}
dest_uuid: "e41b19ce379e45198aeff6fd3814bb43"
cas_config_opid_index: -1
I20260619 14:37:51.352838 6512 consensus_queue.cc:237] T 00000000000000000000000000000000 P e41b19ce379e45198aeff6fd3814bb43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e41b19ce379e45198aeff6fd3814bb43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46665 } } peers { permanent_uuid: "6731bc26510d493c99ecc1aba97fd05b" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39319 } attrs { promote: true } }
I20260619 14:37:51.354539 7070 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e41b19ce379e45198aeff6fd3814bb43 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 6731bc26510d493c99ecc1aba97fd05b (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e41b19ce379e45198aeff6fd3814bb43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46665 } } peers { permanent_uuid: "6731bc26510d493c99ecc1aba97fd05b" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39319 } attrs { promote: true } } }
I20260619 14:37:51.355406 7018 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:51.356817 7018 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b [term 1 FOLLOWER]: Refusing update from remote peer e41b19ce379e45198aeff6fd3814bb43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260619 14:37:51.357471 7071 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e41b19ce379e45198aeff6fd3814bb43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6731bc26510d493c99ecc1aba97fd05b" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39319 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:37:51.358446 7070 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:39319
I20260619 14:37:51.358740 7071 sys_catalog.cc:455] T 00000000000000000000000000000000 P e41b19ce379e45198aeff6fd3814bb43 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e41b19ce379e45198aeff6fd3814bb43" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e41b19ce379e45198aeff6fd3814bb43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46665 } } peers { permanent_uuid: "6731bc26510d493c99ecc1aba97fd05b" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39319 } attrs { promote: true } } }
I20260619 14:37:51.358832 7071 sys_catalog.cc:458] T 00000000000000000000000000000000 P e41b19ce379e45198aeff6fd3814bb43 [sys.catalog]: This master's current role is: LEADER
I20260619 14:37:51.358963 6976 master.cc:537] Master@127.0.98.61:39319 shutting down...
I20260619 14:37:51.362957 7018 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 6731bc26510d493c99ecc1aba97fd05b (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e41b19ce379e45198aeff6fd3814bb43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46665 } } peers { permanent_uuid: "6731bc26510d493c99ecc1aba97fd05b" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39319 } attrs { promote: true } } }
I20260619 14:37:51.365128 7050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e41b19ce379e45198aeff6fd3814bb43. Latest consensus state: current_term: 1 leader_uuid: "e41b19ce379e45198aeff6fd3814bb43" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e41b19ce379e45198aeff6fd3814bb43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46665 } } peers { permanent_uuid: "6731bc26510d493c99ecc1aba97fd05b" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39319 } attrs { promote: true } } }
I20260619 14:37:51.365242 7050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b [sys.catalog]: This master's current role is: LEARNER
I20260619 14:37:51.365324 7050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e41b19ce379e45198aeff6fd3814bb43" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e41b19ce379e45198aeff6fd3814bb43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46665 } } peers { permanent_uuid: "6731bc26510d493c99ecc1aba97fd05b" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39319 } attrs { promote: true } } }
I20260619 14:37:51.365387 7050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b [sys.catalog]: This master's current role is: LEARNER
W20260619 14:37:51.365357 6543 messenger.cc:188] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.0.98.62:46665
I20260619 14:37:51.365444 7050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e41b19ce379e45198aeff6fd3814bb43" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e41b19ce379e45198aeff6fd3814bb43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46665 } } peers { permanent_uuid: "6731bc26510d493c99ecc1aba97fd05b" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39319 } attrs { promote: true } } }
I20260619 14:37:51.365502 7050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b [sys.catalog]: This master's current role is: LEARNER
I20260619 14:37:51.368614 6976 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b [term 1 LEARNER]: Raft consensus shutting down.
I20260619 14:37:51.368893 6976 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b [term 1 LEARNER]: Raft consensus is shut down!
I20260619 14:37:51.368966 6976 tablet_replica.cc:330] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b: stopping tablet replica
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260619 14:37:51.388906 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39349
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46665,127.0.98.61:39319,127.0.98.60:39349
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:37:51.390202 6976 master.cc:552] Master@127.0.98.61:39319 shutdown complete.
I20260619 14:37:51.390254 6976 master_runner.cc:305] Clearing existing system tablet
I20260619 14:37:51.390450 6976 fs_manager.cc:359] Metadata directory not provided
I20260619 14:37:51.390494 6976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:37:51.391510 6976 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 14:37:51.392235 7085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:51.392444 6976 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:37:51.392565 6976 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 14:37:51.392602 6976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-1/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-1/wal
uuid: "6731bc26510d493c99ecc1aba97fd05b"
format_stamp: "Formatted at 2026-06-19 14:37:51 on dist-test-slave-32hj"
server_key: "592f95d9f9a2d16e26653fc44bb8bdac"
server_key_iv: "c935ef08b03e421e200492e18742fd06"
server_key_version: "encryptionkey@0"
I20260619 14:37:51.394011 6976 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:37:51.395726 6976 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 14:37:51.395800 6976 log.cc:1196] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b: Deleting WAL directory at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260619 14:37:51.396062 6976 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b: Deleting consensus metadata
I20260619 14:37:51.396204 6976 master_runner.cc:315] Copying system tablet from 127.0.98.62:46665
I20260619 14:37:51.397351 6976 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:46665
I20260619 14:37:51.407827 6551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:51.401146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48342 (local address 127.0.98.62:46665)
0619 14:37:51.401271 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:51.401276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:51.401493 (+ 217us) server_negotiation.cc:407] Connection header received
0619 14:37:51.401667 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:51.401670 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:51.401717 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:51.401786 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:51.402320 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.402898 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.403681 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.403906 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.403993 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:51.405056 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:51.405060 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:51.405063 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:51.405087 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:51.406367 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:51.406741 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:51.406744 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:51.406746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:51.406823 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:51.407057 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:51.407060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:51.407062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:51.407241 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:51.407326 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:51.407569 (+ 243us) server_negotiation.cc:299] Negotiation successful
0619 14:37:51.407674 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 14:37:51.407835 7091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:51.401033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46665 (local address 127.0.0.1:48342)
0619 14:37:51.401378 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:51.401388 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:51.401570 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:51.401811 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:51.401814 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:51.402000 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:51.402177 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:51.402187 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.403000 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.403003 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:51.403573 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:51.403581 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.404001 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.404004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:51.404092 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:51.404549 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:51.404563 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:51.404928 (+ 365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:51.406463 (+ 1535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:51.406467 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:51.406469 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:51.406655 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:51.406901 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:51.406904 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:51.406906 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:51.406982 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:51.407326 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:51.407329 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:51.407420 (+ 91us) client_negotiation.cc:769] Sending connection context
0619 14:37:51.407503 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 14:37:51.407638 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":86,"threads_started":1}
I20260619 14:37:51.408624 6532 tablet_copy_service.cc:140] P e41b19ce379e45198aeff6fd3814bb43: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 6731bc26510d493c99ecc1aba97fd05b ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:48342)
I20260619 14:37:51.408802 6532 tablet_copy_service.cc:161] P e41b19ce379e45198aeff6fd3814bb43: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 6731bc26510d493c99ecc1aba97fd05b at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:48342: session id = 6731bc26510d493c99ecc1aba97fd05b-00000000000000000000000000000000
I20260619 14:37:51.410179 6532 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P e41b19ce379e45198aeff6fd3814bb43: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 14:37:51.411515 6976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 14:37:51.413352 6976 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b: tablet copy: Starting download of 0 data blocks...
I20260619 14:37:51.413516 6976 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b: tablet copy: Starting download of 1 WAL segments...
I20260619 14:37:51.414985 6976 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 14:37:51.416419 6532 tablet_copy_service.cc:342] P e41b19ce379e45198aeff6fd3814bb43: Request end of tablet copy session 6731bc26510d493c99ecc1aba97fd05b-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:48342
I20260619 14:37:51.416512 6532 tablet_copy_service.cc:433] P e41b19ce379e45198aeff6fd3814bb43: ending tablet copy session 6731bc26510d493c99ecc1aba97fd05b-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 6731bc26510d493c99ecc1aba97fd05b
I20260619 14:37:51.422819 6976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:37:51.424400 7092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:51.424347 7095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:51.424751 6976 server_base.cc:1034] running on GCE node
W20260619 14:37:51.424788 7093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:51.425050 6976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:37:51.425313 6976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:37:51.426442 6976 hybrid_clock.cc:648] HybridClock initialized: now 1781879871426435 us; error 24 us; skew 500 ppm
Jun 19 14:37:51 dist-test-slave-32hj krb5kdc[6467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879871, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:37:51.427886 6976 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260619 14:37:51.428344 6976 webserver.cc:460] Webserver started at http://127.0.98.61:45281/ using document root <none> and password file <none>
I20260619 14:37:51.428526 6976 fs_manager.cc:359] Metadata directory not provided
I20260619 14:37:51.428568 6976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:37:51.429634 6976 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 14:37:51.430387 7102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:51.430594 6976 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 14:37:51.430658 6976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-1/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-1/wal
uuid: "6731bc26510d493c99ecc1aba97fd05b"
format_stamp: "Formatted at 2026-06-19 14:37:51 on dist-test-slave-32hj"
server_key: "592f95d9f9a2d16e26653fc44bb8bdac"
server_key_iv: "c935ef08b03e421e200492e18742fd06"
server_key_version: "encryptionkey@0"
I20260619 14:37:51.430730 6976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:37:51.442463 6976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:37:51.444162 6976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:37:51.448518 6976 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:39319
I20260619 14:37:51.448530 7154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:39319 every 8 connection(s)
I20260619 14:37:51.449072 6976 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-1/data/info.pb
I20260619 14:37:51.450464 7155 sys_catalog.cc:263] Verifying existing consensus state
I20260619 14:37:51.451253 7155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b: Bootstrap starting.
I20260619 14:37:51.456995 7155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:37:51.457258 7155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b: Bootstrap complete.
I20260619 14:37:51.457757 7155 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e41b19ce379e45198aeff6fd3814bb43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46665 } } peers { permanent_uuid: "6731bc26510d493c99ecc1aba97fd05b" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39319 } attrs { promote: true } }
I20260619 14:37:51.457998 7155 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6731bc26510d493c99ecc1aba97fd05b, State: Initialized, Role: LEARNER
I20260619 14:37:51.458138 7155 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e41b19ce379e45198aeff6fd3814bb43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46665 } } peers { permanent_uuid: "6731bc26510d493c99ecc1aba97fd05b" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39319 } attrs { promote: true } }
I20260619 14:37:51.458436 7158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e41b19ce379e45198aeff6fd3814bb43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46665 } } peers { permanent_uuid: "6731bc26510d493c99ecc1aba97fd05b" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39319 } attrs { promote: true } } }
I20260619 14:37:51.458583 7155 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:37:51.458586 7158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b [sys.catalog]: This master's current role is: LEARNER
I20260619 14:37:51.461314 7169 catalog_manager.cc:1259] Loaded cluster ID: a546ae03a7f4400d99fa082375d31b43
I20260619 14:37:51.461429 7169 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b: loading cluster ID for follower catalog manager: success
I20260619 14:37:51.462980 7169 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b: acquiring CA information for follower catalog manager: success
I20260619 14:37:51.464047 7169 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 14:37:51.532173 7083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:37:51.532423 7083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:37:51.532462 7083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:37:51.536245 7083 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:37:51.536293 7083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:37:51.536311 7083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:37:51.536326 7083 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:37:51.536339 7083 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:37:51.541309 7083 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46665,127.0.98.61:39319,127.0.98.60:39349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:37:51.542240 7083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:37:51.543141 7083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:37:51.550892 7175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:51.550900 7177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:51.550976 7083 server_base.cc:1034] running on GCE node
W20260619 14:37:51.550899 7174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:51.551782 7083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:37:51.552855 7083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:37:51.553996 7083 hybrid_clock.cc:648] HybridClock initialized: now 1781879871553972 us; error 40 us; skew 500 ppm
Jun 19 14:37:51 dist-test-slave-32hj krb5kdc[6467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879871, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:37:51.556456 7083 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260619 14:37:51.557277 7083 webserver.cc:460] Webserver started at http://127.0.98.60:37961/ using document root <none> and password file <none>
I20260619 14:37:51.557766 7083 fs_manager.cc:359] Metadata directory not provided
I20260619 14:37:51.557829 7083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:37:51.557986 7083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:37:51.560149 7083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-2/data/instance:
uuid: "827bf26a4cd440e59c8db36bbc95d991"
format_stamp: "Formatted at 2026-06-19 14:37:51 on dist-test-slave-32hj"
server_key: "63519ad0a850a4fa8f8362462bc9d8ba"
server_key_iv: "e2e35d5acbf17c68aeabcff52ddf6428"
server_key_version: "encryptionkey@0"
I20260619 14:37:51.560663 7083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-2/wal/instance:
uuid: "827bf26a4cd440e59c8db36bbc95d991"
format_stamp: "Formatted at 2026-06-19 14:37:51 on dist-test-slave-32hj"
server_key: "63519ad0a850a4fa8f8362462bc9d8ba"
server_key_iv: "e2e35d5acbf17c68aeabcff52ddf6428"
server_key_version: "encryptionkey@0"
I20260619 14:37:51.564086 7083 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:37:51.566288 7184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:51.567399 7083 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:37:51.567495 7083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-2/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-2/wal
uuid: "827bf26a4cd440e59c8db36bbc95d991"
format_stamp: "Formatted at 2026-06-19 14:37:51 on dist-test-slave-32hj"
server_key: "63519ad0a850a4fa8f8362462bc9d8ba"
server_key_iv: "e2e35d5acbf17c68aeabcff52ddf6428"
server_key_version: "encryptionkey@0"
I20260619 14:37:51.567588 7083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:37:51.576187 7083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:37:51.579196 7083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:37:51.579353 7083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:37:51.586182 7083 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:39349
I20260619 14:37:51.586436 7236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:39349 every 8 connection(s)
I20260619 14:37:51.587064 7083 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-2/data/info.pb
I20260619 14:37:51.589682 7237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:51.593257 7237 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46665 } has no permanent_uuid. Determining permanent_uuid...
W20260619 14:37:51.594950 6547 acceptor_pool.cc:285] unable to collect diagnostics on RPC socket 127.0.98.62:46665: Invalid argument: socket address is ambiguous
I20260619 14:37:51.596199 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 7083
Jun 19 14:37:51 dist-test-slave-32hj krb5kdc[6467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879871, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 19 14:37:51 dist-test-slave-32hj krb5kdc[6467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260619 14:37:51.606325 6551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:51.595137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48344 (local address 127.0.98.62:46665)
0619 14:37:51.595251 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:51.595255 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:51.595908 (+ 653us) server_negotiation.cc:407] Connection header received
0619 14:37:51.597058 (+ 1150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:51.597063 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:51.597122 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:51.597177 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:51.598385 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.598889 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.599839 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.600049 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.600141 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:51.602656 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:51.602660 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:51.602663 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:51.602687 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:51.604185 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:51.604745 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:51.604749 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:51.604751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:51.604843 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:51.605220 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:51.605224 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:51.605226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:51.605461 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:51.605550 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:51.606034 (+ 484us) server_negotiation.cc:299] Negotiation successful
0619 14:37:51.606166 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 14:37:51.607131 7240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:51.597018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57520 (local address 127.0.98.60:39349)
0619 14:37:51.597283 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:51.597294 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:51.597332 (+ 38us) server_negotiation.cc:407] Connection header received
0619 14:37:51.597449 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:51.597453 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:51.597608 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:51.597753 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:51.598478 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.599320 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.600073 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.600424 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.600509 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:51.603074 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:51.603081 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:51.603092 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:51.603115 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:51.604663 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:51.605220 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:51.605226 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:51.605228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:51.605317 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:51.605585 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:51.605589 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:51.605591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:51.605850 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:51.606028 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:51.606222 (+ 194us) server_negotiation.cc:299] Negotiation successful
0619 14:37:51.606354 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":175,"spinlock_wait_cycles":43648,"thread_start_us":89,"threads_started":1}
I20260619 14:37:51.607137 7239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:51.595352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46665 (local address 127.0.0.1:48344)
0619 14:37:51.595754 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:51.595794 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:51.596814 (+ 1020us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:51.597472 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:51.597479 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:51.597786 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:51.598248 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:51.598260 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.599002 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.599005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:51.599726 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:51.599735 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.600175 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.600179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:51.600376 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:51.600973 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:51.600994 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:51.602511 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:51.604301 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:51.604308 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:51.604315 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:51.604627 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:51.604928 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:51.604931 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:51.604933 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:51.605092 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:51.605562 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:51.605569 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:51.605813 (+ 244us) client_negotiation.cc:769] Sending connection context
0619 14:37:51.605969 (+ 156us) client_negotiation.cc:240] Negotiation successful
0619 14:37:51.606275 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":39,"thread_start_us":91,"threads_started":1}
I20260619 14:37:51.608712 7237 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39319 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:37:51 dist-test-slave-32hj krb5kdc[6467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879871, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 14:37:51.616580 7239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:51.609238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39319 (local address 127.0.0.1:47434)
0619 14:37:51.609378 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:51.609392 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:51.609467 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:51.610275 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:51.610278 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:51.610414 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:51.610499 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:51.610504 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.611338 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.611340 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:51.611844 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:51.611850 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.612277 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.612279 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:51.612363 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:51.612784 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:51.612795 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:51.613936 (+ 1141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:51.615481 (+ 1545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:51.615483 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:51.615485 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:51.615653 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:51.615892 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:51.615894 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:51.615896 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:51.615959 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:51.616240 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:51.616242 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:51.616309 (+ 67us) client_negotiation.cc:769] Sending connection context
0619 14:37:51.616379 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 14:37:51.616467 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 14:37:51.616628 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:51.609343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47434 (local address 127.0.98.61:39319)
0619 14:37:51.610071 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:51.610074 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:51.610087 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:37:51.610121 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:51.610123 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:51.610172 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:51.610246 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:51.610586 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.611249 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.611957 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.612192 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.612269 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:51.614107 (+ 1838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:51.614111 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:51.614113 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:51.614136 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:51.615398 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:51.615727 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:51.615729 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:51.615731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:51.615802 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:51.616020 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:51.616022 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:51.616023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:51.616178 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:51.616248 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:51.616418 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 14:37:51.616511 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":532,"threads_started":1}
I20260619 14:37:51.617317 7237 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39349 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:37:51 dist-test-slave-32hj krb5kdc[6467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879871, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260619 14:37:51.623997 7239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:51.617705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39349 (local address 127.0.0.1:57536)
0619 14:37:51.617819 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:51.617838 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:51.617904 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:51.618069 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:51.618071 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:51.618190 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:51.618254 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:51.618258 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.618916 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.618919 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:51.619413 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:51.619420 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.619818 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.619839 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:51.619899 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:51.620283 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:51.620294 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:51.621409 (+ 1115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:51.622774 (+ 1365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:51.622776 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:51.622779 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:51.622950 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:51.623177 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:51.623179 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:51.623181 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:51.623246 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:51.623547 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:51.623550 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:51.623623 (+ 73us) client_negotiation.cc:769] Sending connection context
0619 14:37:51.623708 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 14:37:51.623826 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 14:37:51.624001 7240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:51.617787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57536 (local address 127.0.98.60:39349)
0619 14:37:51.617871 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:51.617875 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:51.617888 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:37:51.617963 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:51.617965 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:51.617996 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:51.618052 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:51.618351 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.618819 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.619496 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.619696 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.619760 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:51.621519 (+ 1759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:51.621523 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:51.621525 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:51.621544 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:51.622670 (+ 1126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:51.623030 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:51.623034 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:51.623036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:51.623100 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:51.623318 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:51.623321 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:51.623323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:51.623468 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:51.623546 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:51.623758 (+ 212us) server_negotiation.cc:299] Negotiation successful
0619 14:37:51.623878 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":24}
I20260619 14:37:51.627185 7237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991: Bootstrap starting.
I20260619 14:37:51.629459 7237 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:51.630151 7237 log.cc:824] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991: Log is configured to *not* fsync() on all Append() calls
I20260619 14:37:51.632331 7237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991: No bootstrap required, opened a new log
I20260619 14:37:51.635848 7237 raft_consensus.cc:348] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e41b19ce379e45198aeff6fd3814bb43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46665 } } peers { permanent_uuid: "6731bc26510d493c99ecc1aba97fd05b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39319 } } peers { permanent_uuid: "827bf26a4cd440e59c8db36bbc95d991" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39349 } }
I20260619 14:37:51.636053 7237 raft_consensus.cc:374] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:51.636086 7237 raft_consensus.cc:729] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 827bf26a4cd440e59c8db36bbc95d991, State: Initialized, Role: FOLLOWER
I20260619 14:37:51.636451 7237 consensus_queue.cc:260] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e41b19ce379e45198aeff6fd3814bb43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46665 } } peers { permanent_uuid: "6731bc26510d493c99ecc1aba97fd05b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39319 } } peers { permanent_uuid: "827bf26a4cd440e59c8db36bbc95d991" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39349 } }
I20260619 14:37:51.636893 7243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e41b19ce379e45198aeff6fd3814bb43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46665 } } peers { permanent_uuid: "6731bc26510d493c99ecc1aba97fd05b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39319 } } peers { permanent_uuid: "827bf26a4cd440e59c8db36bbc95d991" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39349 } } }
I20260619 14:37:51.637008 7243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:37:51.637537 7237 sys_catalog.cc:564] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 14:37:51.641938 7254 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:37:51.642016 7254 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:37:51.644867 7083 master_runner.cc:428] Detected that this master 827bf26a4cd440e59c8db36bbc95d991 is joining an existing cluster
I20260619 14:37:51.644915 7083 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:39349
I20260619 14:37:51.653147 6703 heartbeater.cc:503] Master 127.0.98.62:46665 was elected leader, sending a full tablet report...
I20260619 14:37:51.654322 7240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:51.647488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57542 (local address 127.0.98.60:39349)
0619 14:37:51.647583 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:51.647587 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:51.647681 (+ 94us) server_negotiation.cc:407] Connection header received
0619 14:37:51.647766 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:51.647769 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:51.647820 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:51.647898 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:51.648280 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.648977 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.650021 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.650345 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.650434 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:51.651630 (+ 1196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:51.651634 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:51.651636 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:51.651656 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:51.652829 (+ 1173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:51.653235 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:51.653238 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:51.653240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:51.653325 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:51.653567 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:51.653571 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:51.653572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:51.653746 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:51.653828 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:51.654052 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 14:37:51.654176 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":39}
I20260619 14:37:51.654330 7262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:51.646822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39349 (local address 127.0.0.1:57542)
0619 14:37:51.647596 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:51.647608 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:51.647692 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:51.647919 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:51.647922 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:51.648091 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:51.648184 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:51.648191 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.649071 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.649074 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:51.649926 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:51.649934 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.650500 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.650503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:51.650578 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:51.651059 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:51.651073 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:51.651529 (+ 456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:51.652910 (+ 1381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:51.652913 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:51.652915 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:51.653139 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:51.653421 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:51.653423 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:51.653425 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:51.653498 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:51.653829 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:51.653831 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:51.653908 (+ 77us) client_negotiation.cc:769] Sending connection context
0619 14:37:51.653998 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 14:37:51.654128 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"thread_start_us":98,"threads_started":1}
I20260619 14:37:51.660497 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 6574
I20260619 14:37:51.666118 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:33487
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:46665
--webserver_port=33719
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:46665,127.0.98.61:39319
--tserver_master_addrs=127.0.98.62:46665,127.0.98.61:39319,127.0.98.60:39349 with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:37:51.677450 6551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:51.646850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48354 (local address 127.0.98.62:46665)
0619 14:37:51.647186 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:51.647190 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:51.647203 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:37:51.647239 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:51.647252 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:51.647296 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:51.647363 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:51.648098 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.648948 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.650667 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.650977 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.651048 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:51.673628 (+ 22580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:51.673633 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:51.673636 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:51.673669 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:51.675683 (+ 2014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:51.676155 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:51.676159 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:51.676161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:51.676237 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:51.676505 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:51.676509 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:51.676510 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:51.676759 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:51.676838 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:51.677173 (+ 335us) server_negotiation.cc:299] Negotiation successful
0619 14:37:51.677287 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259}
I20260619 14:37:51.677619 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:51.647246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47444 (local address 127.0.98.61:39319)
0619 14:37:51.647369 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:51.647372 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:51.648253 (+ 881us) server_negotiation.cc:407] Connection header received
0619 14:37:51.648394 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:51.648397 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:51.648443 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:51.648488 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:51.649069 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.649843 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.650821 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.651117 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.651209 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:51.674351 (+ 23142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:51.674356 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:51.674358 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:51.674395 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:51.675915 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:51.676372 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:51.676376 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:51.676378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:51.676451 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:51.676745 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:51.676748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:51.676750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:51.676958 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:51.677021 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:51.677347 (+ 326us) server_negotiation.cc:299] Negotiation successful
0619 14:37:51.677467 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 14:37:51.678125 7261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:51.647362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39319 (local address 127.0.0.1:47444)
0619 14:37:51.648173 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:51.648184 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:51.648305 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:51.648606 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:51.648609 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:51.648847 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:51.648946 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:51.648952 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.649964 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.649967 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:51.650738 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:51.650746 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.651313 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.651317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:51.651398 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:51.673704 (+ 22306us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:51.673722 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:51.674154 (+ 432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:51.676039 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:51.676043 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:51.676045 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:51.676270 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:51.676548 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:51.676551 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:51.676553 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:51.676637 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:51.677089 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:51.677091 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:51.677188 (+ 97us) client_negotiation.cc:769] Sending connection context
0619 14:37:51.677798 (+ 610us) client_negotiation.cc:240] Negotiation successful
0619 14:37:51.677950 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":749,"thread_start_us":78,"threads_started":1}
I20260619 14:37:51.678530 7260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:51.646803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46665 (local address 127.0.0.1:48354)
0619 14:37:51.647047 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:51.647061 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:51.647147 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:51.647637 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:51.647639 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:51.647860 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:51.647986 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:51.647993 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.649074 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.649078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:51.650546 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:51.650552 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.651188 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.651191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:51.651282 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:51.651678 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:51.651692 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:51.673440 (+ 21748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:51.675812 (+ 2372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:51.675815 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:51.675818 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:51.676040 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:51.676339 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:51.676342 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:51.676344 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:51.676418 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:51.676903 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:51.676905 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:51.677008 (+ 103us) client_negotiation.cc:769] Sending connection context
0619 14:37:51.677987 (+ 979us) client_negotiation.cc:240] Negotiation successful
0619 14:37:51.678355 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":26,"thread_start_us":92,"threads_started":1}
Jun 19 14:37:51 dist-test-slave-32hj krb5kdc[6467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879870, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260619 14:37:51.690347 7069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:51.682184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39349 (local address 127.0.0.1:57552)
0619 14:37:51.682292 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:51.682317 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:51.682400 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:51.682663 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:51.682666 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:51.682843 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:51.682938 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:51.682945 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.683870 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.683873 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:51.684747 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:51.684757 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.685277 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.685280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:51.685373 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:51.685778 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:51.685791 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:51.687202 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:51.688944 (+ 1742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:51.688947 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:51.688949 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:51.689171 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:51.689445 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:51.689449 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:51.689451 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:51.689541 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:51.689897 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:51.689899 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:51.689987 (+ 88us) client_negotiation.cc:769] Sending connection context
0619 14:37:51.690085 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 14:37:51.690204 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260619 14:37:51.690362 7240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:51.682233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57552 (local address 127.0.98.60:39349)
0619 14:37:51.682320 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:51.682324 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:51.682360 (+ 36us) server_negotiation.cc:407] Connection header received
0619 14:37:51.682511 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:51.682514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:51.682562 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:51.682630 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:51.683036 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.683769 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.684854 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.685177 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.685271 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:51.687328 (+ 2057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:51.687332 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:51.687334 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:51.687357 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:51.688832 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:51.689276 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:51.689279 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:51.689281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:51.689354 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:51.689623 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:51.689626 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:51.689628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:51.689813 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:51.689899 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:51.690132 (+ 233us) server_negotiation.cc:299] Negotiation successful
0619 14:37:51.690226 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 14:37:51.691565 6512 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:39349: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "827bf26a4cd440e59c8db36bbc95d991"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 39349
}
attrs {
promote: true
}
}
dest_uuid: "e41b19ce379e45198aeff6fd3814bb43"
cas_config_opid_index: 5
I20260619 14:37:51.692128 6512 consensus_queue.cc:237] T 00000000000000000000000000000000 P e41b19ce379e45198aeff6fd3814bb43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e41b19ce379e45198aeff6fd3814bb43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46665 } } peers { permanent_uuid: "6731bc26510d493c99ecc1aba97fd05b" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39319 } attrs { promote: true } } peers { permanent_uuid: "827bf26a4cd440e59c8db36bbc95d991" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39349 } attrs { promote: true } }
I20260619 14:37:51.693128 7073 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e41b19ce379e45198aeff6fd3814bb43 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 827bf26a4cd440e59c8db36bbc95d991 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e41b19ce379e45198aeff6fd3814bb43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46665 } } peers { permanent_uuid: "6731bc26510d493c99ecc1aba97fd05b" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39319 } attrs { promote: true } } peers { permanent_uuid: "827bf26a4cd440e59c8db36bbc95d991" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39349 } attrs { promote: true } } }
I20260619 14:37:51.694528 7073 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:39349
I20260619 14:37:51.694672 7070 sys_catalog.cc:455] T 00000000000000000000000000000000 P e41b19ce379e45198aeff6fd3814bb43 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e41b19ce379e45198aeff6fd3814bb43" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e41b19ce379e45198aeff6fd3814bb43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46665 } } peers { permanent_uuid: "6731bc26510d493c99ecc1aba97fd05b" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39319 } attrs { promote: true } } peers { permanent_uuid: "827bf26a4cd440e59c8db36bbc95d991" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39349 } attrs { promote: true } } }
I20260619 14:37:51.694772 7070 sys_catalog.cc:458] T 00000000000000000000000000000000 P e41b19ce379e45198aeff6fd3814bb43 [sys.catalog]: This master's current role is: LEADER
I20260619 14:37:51.696277 7211 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:51.697589 7211 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991 [term 1 FOLLOWER]: Refusing update from remote peer e41b19ce379e45198aeff6fd3814bb43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 14:37:51.697960 7083 master.cc:537] Master@127.0.98.60:39349 shutting down...
I20260619 14:37:51.698516 7070 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e41b19ce379e45198aeff6fd3814bb43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "827bf26a4cd440e59c8db36bbc95d991" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39349 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 14:37:51.699622 6497 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260619 14:37:51.703256 6497 consensus_peers.cc:487] T 00000000000000000000000000000000 P e41b19ce379e45198aeff6fd3814bb43 -> Peer 827bf26a4cd440e59c8db36bbc95d991 (127.0.98.60:39349): Couldn't send request to peer 827bf26a4cd440e59c8db36bbc95d991. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 14:37:51.704068 7083 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:37:51.704329 7083 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:37:51.704428 7083 tablet_replica.cc:330] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991: stopping tablet replica
I20260619 14:37:51.704653 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:51.695660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47448 (local address 127.0.98.61:39319)
0619 14:37:51.695770 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:51.695775 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:51.700078 (+ 4303us) server_negotiation.cc:407] Connection header received
0619 14:37:51.700131 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:51.700134 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:51.700191 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:51.700283 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:37:51.700538 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.701337 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.702790 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.703421 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.703494 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:51.703627 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:51.704458 (+ 831us) server_negotiation.cc:299] Negotiation successful
0619 14:37:51.704534 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 14:37:51.705148 7069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:51.694848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39319 (local address 127.0.0.1:47448)
0619 14:37:51.696063 (+ 1215us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:51.696077 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:51.696146 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:51.700316 (+ 4170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:51.700319 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:51.700337 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:37:51.700414 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:51.700420 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.701458 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.701461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:51.702658 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:51.702668 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.704075 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.704078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:51.704229 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:51.704271 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 14:37:51.704341 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 14:37:51.705040 (+ 699us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1152}
I20260619 14:37:51.705713 7129 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b [term 1 LEARNER]: Refusing update from remote peer e41b19ce379e45198aeff6fd3814bb43: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 14:37:51.706079 7070 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e41b19ce379e45198aeff6fd3814bb43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6731bc26510d493c99ecc1aba97fd05b" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39319 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260619 14:37:51.707468 7129 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 827bf26a4cd440e59c8db36bbc95d991 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e41b19ce379e45198aeff6fd3814bb43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46665 } } peers { permanent_uuid: "6731bc26510d493c99ecc1aba97fd05b" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39319 } attrs { promote: true } } peers { permanent_uuid: "827bf26a4cd440e59c8db36bbc95d991" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39349 } attrs { promote: true } } }
I20260619 14:37:51.709072 7266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e41b19ce379e45198aeff6fd3814bb43" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e41b19ce379e45198aeff6fd3814bb43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46665 } } peers { permanent_uuid: "6731bc26510d493c99ecc1aba97fd05b" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39319 } attrs { promote: true } } peers { permanent_uuid: "827bf26a4cd440e59c8db36bbc95d991" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39349 } attrs { promote: true } } }
I20260619 14:37:51.709142 7158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e41b19ce379e45198aeff6fd3814bb43. Latest consensus state: current_term: 1 leader_uuid: "e41b19ce379e45198aeff6fd3814bb43" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e41b19ce379e45198aeff6fd3814bb43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46665 } } peers { permanent_uuid: "6731bc26510d493c99ecc1aba97fd05b" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39319 } attrs { promote: true } } peers { permanent_uuid: "827bf26a4cd440e59c8db36bbc95d991" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39349 } attrs { promote: true } } }
I20260619 14:37:51.709262 7158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b [sys.catalog]: This master's current role is: LEARNER
I20260619 14:37:51.709381 7266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b [sys.catalog]: This master's current role is: LEARNER
I20260619 14:37:51.710091 7070 raft_consensus.cc:1050] T 00000000000000000000000000000000 P e41b19ce379e45198aeff6fd3814bb43: attempting to promote NON_VOTER 6731bc26510d493c99ecc1aba97fd05b to VOTER
I20260619 14:37:51.710512 7070 consensus_queue.cc:237] T 00000000000000000000000000000000 P e41b19ce379e45198aeff6fd3814bb43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e41b19ce379e45198aeff6fd3814bb43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46665 } } peers { permanent_uuid: "6731bc26510d493c99ecc1aba97fd05b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39319 } attrs { promote: false } } peers { permanent_uuid: "827bf26a4cd440e59c8db36bbc95d991" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39349 } attrs { promote: true } }
I20260619 14:37:51.711591 7129 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b [term 1 LEARNER]: Refusing update from remote peer e41b19ce379e45198aeff6fd3814bb43: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260619 14:37:51.711694 7069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:51.711420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39349 (local address 127.0.0.1:57556)
0619 14:37:51.711565 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:51.711625 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260619 14:37:51.712011 7071 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e41b19ce379e45198aeff6fd3814bb43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6731bc26510d493c99ecc1aba97fd05b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260619 14:37:51.712363 7069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:51.712133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39349 (local address 127.0.0.1:57566)
0619 14:37:51.712247 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:51.712288 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260619 14:37:51.712479 6497 consensus_peers.cc:487] T 00000000000000000000000000000000 P e41b19ce379e45198aeff6fd3814bb43 -> Peer 827bf26a4cd440e59c8db36bbc95d991 (127.0.98.60:39349): Couldn't send request to peer 827bf26a4cd440e59c8db36bbc95d991. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39349: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 14:37:51.713791 7073 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e41b19ce379e45198aeff6fd3814bb43 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 6731bc26510d493c99ecc1aba97fd05b (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e41b19ce379e45198aeff6fd3814bb43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46665 } } peers { permanent_uuid: "6731bc26510d493c99ecc1aba97fd05b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39319 } attrs { promote: false } } peers { permanent_uuid: "827bf26a4cd440e59c8db36bbc95d991" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39349 } attrs { promote: true } } }
I20260619 14:37:51.714272 7129 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 6731bc26510d493c99ecc1aba97fd05b (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e41b19ce379e45198aeff6fd3814bb43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46665 } } peers { permanent_uuid: "6731bc26510d493c99ecc1aba97fd05b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39319 } attrs { promote: false } } peers { permanent_uuid: "827bf26a4cd440e59c8db36bbc95d991" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39349 } attrs { promote: true } } }
I20260619 14:37:51.715139 7073 sys_catalog.cc:455] T 00000000000000000000000000000000 P e41b19ce379e45198aeff6fd3814bb43 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e41b19ce379e45198aeff6fd3814bb43" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e41b19ce379e45198aeff6fd3814bb43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46665 } } peers { permanent_uuid: "6731bc26510d493c99ecc1aba97fd05b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39319 } attrs { promote: false } } peers { permanent_uuid: "827bf26a4cd440e59c8db36bbc95d991" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39349 } attrs { promote: true } } }
I20260619 14:37:51.715229 7073 sys_catalog.cc:458] T 00000000000000000000000000000000 P e41b19ce379e45198aeff6fd3814bb43 [sys.catalog]: This master's current role is: LEADER
I20260619 14:37:51.715325 7073 sys_catalog.cc:455] T 00000000000000000000000000000000 P e41b19ce379e45198aeff6fd3814bb43 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e41b19ce379e45198aeff6fd3814bb43" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e41b19ce379e45198aeff6fd3814bb43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46665 } } peers { permanent_uuid: "6731bc26510d493c99ecc1aba97fd05b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39319 } attrs { promote: false } } peers { permanent_uuid: "827bf26a4cd440e59c8db36bbc95d991" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39349 } attrs { promote: true } } }
I20260619 14:37:51.715386 7073 sys_catalog.cc:458] T 00000000000000000000000000000000 P e41b19ce379e45198aeff6fd3814bb43 [sys.catalog]: This master's current role is: LEADER
I20260619 14:37:51.716118 7266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e41b19ce379e45198aeff6fd3814bb43" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e41b19ce379e45198aeff6fd3814bb43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46665 } } peers { permanent_uuid: "6731bc26510d493c99ecc1aba97fd05b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39319 } attrs { promote: false } } peers { permanent_uuid: "827bf26a4cd440e59c8db36bbc95d991" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39349 } attrs { promote: true } } }
I20260619 14:37:51.716257 7266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:37:51.719568 7083 master.cc:552] Master@127.0.98.60:39349 shutdown complete.
I20260619 14:37:51.719619 7083 master_runner.cc:305] Clearing existing system tablet
I20260619 14:37:51.719847 7083 fs_manager.cc:359] Metadata directory not provided
I20260619 14:37:51.719902 7083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:37:51.721179 7083 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 14:37:51.721993 7268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:51.722221 7083 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:37:51.722343 7083 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 14:37:51.722381 7083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-2/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-2/wal
uuid: "827bf26a4cd440e59c8db36bbc95d991"
format_stamp: "Formatted at 2026-06-19 14:37:51 on dist-test-slave-32hj"
server_key: "63519ad0a850a4fa8f8362462bc9d8ba"
server_key_iv: "e2e35d5acbf17c68aeabcff52ddf6428"
server_key_version: "encryptionkey@0"
I20260619 14:37:51.724253 7083 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:37:51.726516 7083 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 14:37:51.726598 7083 log.cc:1196] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991: Deleting WAL directory at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260619 14:37:51.726809 7083 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991: Deleting consensus metadata
I20260619 14:37:51.726972 7083 master_runner.cc:315] Copying system tablet from 127.0.98.62:46665
I20260619 14:37:51.728013 7083 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:46665
I20260619 14:37:51.736650 7274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:51.728497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46665 (local address 127.0.0.1:48362)
0619 14:37:51.728859 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:51.728872 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:51.728976 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:51.729218 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:51.729221 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:51.729424 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:51.729523 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:51.729530 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.730541 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.730544 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:51.731644 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:51.731655 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.732222 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.732226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:51.732314 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:51.732921 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:51.732935 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:51.733343 (+ 408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:51.735158 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:51.735162 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:51.735164 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:51.735404 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:51.735674 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:51.735677 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:51.735679 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:51.735765 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:51.736148 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:51.736152 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:51.736248 (+ 96us) client_negotiation.cc:769] Sending connection context
0619 14:37:51.736348 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 14:37:51.736480 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":76,"threads_started":1}
I20260619 14:37:51.736660 6551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:51.728591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48362 (local address 127.0.98.62:46665)
0619 14:37:51.728714 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:51.728752 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:51.728945 (+ 193us) server_negotiation.cc:407] Connection header received
0619 14:37:51.729053 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:51.729056 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:51.729107 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:51.729189 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:51.729633 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.730437 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.731766 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.732111 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.732212 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:51.733485 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:51.733489 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:51.733492 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:51.733517 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:51.735056 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:51.735497 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:51.735501 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:51.735503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:51.735588 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:51.735852 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:51.735855 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:51.735857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:51.736057 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:51.736147 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:51.736405 (+ 258us) server_negotiation.cc:299] Negotiation successful
0619 14:37:51.736515 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 14:37:51.737500 6532 tablet_copy_service.cc:140] P e41b19ce379e45198aeff6fd3814bb43: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 827bf26a4cd440e59c8db36bbc95d991 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:48362)
I20260619 14:37:51.737586 6532 tablet_copy_service.cc:161] P e41b19ce379e45198aeff6fd3814bb43: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 827bf26a4cd440e59c8db36bbc95d991 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:48362: session id = 827bf26a4cd440e59c8db36bbc95d991-00000000000000000000000000000000
I20260619 14:37:51.738677 6532 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P e41b19ce379e45198aeff6fd3814bb43: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 14:37:51.740118 7083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 14:37:51.742965 7083 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991: tablet copy: Starting download of 0 data blocks...
I20260619 14:37:51.743227 7083 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991: tablet copy: Starting download of 1 WAL segments...
I20260619 14:37:51.744622 7083 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 14:37:51.746280 6532 tablet_copy_service.cc:342] P e41b19ce379e45198aeff6fd3814bb43: Request end of tablet copy session 827bf26a4cd440e59c8db36bbc95d991-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:48362
I20260619 14:37:51.746354 6532 tablet_copy_service.cc:433] P e41b19ce379e45198aeff6fd3814bb43: ending tablet copy session 827bf26a4cd440e59c8db36bbc95d991-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 827bf26a4cd440e59c8db36bbc95d991
I20260619 14:37:51.753289 7083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:37:51.754941 7275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:51.755081 7278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:51.755674 7083 server_base.cc:1034] running on GCE node
W20260619 14:37:51.756917 7276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:51.757205 7083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:37:51.757505 7083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:37:51.758656 7083 hybrid_clock.cc:648] HybridClock initialized: now 1781879871758643 us; error 28 us; skew 500 ppm
Jun 19 14:37:51 dist-test-slave-32hj krb5kdc[6467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879871, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:37:51.760365 7083 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260619 14:37:51.760866 7083 webserver.cc:460] Webserver started at http://127.0.98.60:34239/ using document root <none> and password file <none>
I20260619 14:37:51.761047 7083 fs_manager.cc:359] Metadata directory not provided
I20260619 14:37:51.761087 7083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:37:51.762375 7083 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 14:37:51.763411 7285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:51.763681 7083 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 14:37:51.763757 7083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-2/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-2/wal
uuid: "827bf26a4cd440e59c8db36bbc95d991"
format_stamp: "Formatted at 2026-06-19 14:37:51 on dist-test-slave-32hj"
server_key: "63519ad0a850a4fa8f8362462bc9d8ba"
server_key_iv: "e2e35d5acbf17c68aeabcff52ddf6428"
server_key_version: "encryptionkey@0"
I20260619 14:37:51.763849 7083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:37:51.785822 7083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:37:51.787842 7083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:37:51.792455 7083 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:39349
I20260619 14:37:51.792919 7337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:39349 every 8 connection(s)
I20260619 14:37:51.793136 7083 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/master-2/data/info.pb
I20260619 14:37:51.794678 7338 sys_catalog.cc:263] Verifying existing consensus state
I20260619 14:37:51.795615 7338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991: Bootstrap starting.
W20260619 14:37:51.803671 7263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:37:51.803900 7263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:37:51.803959 7263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 14:37:51.804352 7338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:37:51.804586 7338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991: Bootstrap complete.
I20260619 14:37:51.805051 7338 raft_consensus.cc:348] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e41b19ce379e45198aeff6fd3814bb43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46665 } } peers { permanent_uuid: "6731bc26510d493c99ecc1aba97fd05b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39319 } attrs { promote: false } } peers { permanent_uuid: "827bf26a4cd440e59c8db36bbc95d991" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39349 } attrs { promote: true } }
I20260619 14:37:51.805159 7338 raft_consensus.cc:729] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 827bf26a4cd440e59c8db36bbc95d991, State: Initialized, Role: LEARNER
I20260619 14:37:51.805251 7338 consensus_queue.cc:260] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e41b19ce379e45198aeff6fd3814bb43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46665 } } peers { permanent_uuid: "6731bc26510d493c99ecc1aba97fd05b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39319 } attrs { promote: false } } peers { permanent_uuid: "827bf26a4cd440e59c8db36bbc95d991" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39349 } attrs { promote: true } }
I20260619 14:37:51.805548 7344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e41b19ce379e45198aeff6fd3814bb43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46665 } } peers { permanent_uuid: "6731bc26510d493c99ecc1aba97fd05b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39319 } attrs { promote: false } } peers { permanent_uuid: "827bf26a4cd440e59c8db36bbc95d991" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39349 } attrs { promote: true } } }
I20260619 14:37:51.805652 7344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991 [sys.catalog]: This master's current role is: LEARNER
I20260619 14:37:51.805688 7338 sys_catalog.cc:564] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:37:51.807978 7355 catalog_manager.cc:1259] Loaded cluster ID: a546ae03a7f4400d99fa082375d31b43
I20260619 14:37:51.808022 7355 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991: loading cluster ID for follower catalog manager: success
W20260619 14:37:51.808454 7263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:37:51.808513 7263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:37:51.808595 7263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:37:51.809577 7355 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991: acquiring CA information for follower catalog manager: success
I20260619 14:37:51.810606 7355 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 14:37:51.814023 7263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:33487
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=33719
--tserver_master_addrs=127.0.98.62:46665,127.0.98.61:39319,127.0.98.60:39349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:37:51.815078 7263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:37:51.816078 7263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:37:51.825196 7357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:51.825208 7358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:51.825196 7360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:51.825716 7263 server_base.cc:1034] running on GCE node
I20260619 14:37:51.826079 7263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:37:51.826673 7263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:37:51.827824 7263 hybrid_clock.cc:648] HybridClock initialized: now 1781879871827804 us; error 49 us; skew 500 ppm
Jun 19 14:37:51 dist-test-slave-32hj krb5kdc[6467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879871, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:37:51.830467 7263 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260619 14:37:51.831337 7263 webserver.cc:460] Webserver started at http://127.0.98.1:33719/ using document root <none> and password file <none>
I20260619 14:37:51.831881 7263 fs_manager.cc:359] Metadata directory not provided
I20260619 14:37:51.831957 7263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:37:51.835932 7263 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:37:51.837984 7367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:51.839161 7263 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:37:51.839254 7263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-0/wal
uuid: "57ffea6d72ae469491ce019f5e03a885"
format_stamp: "Formatted at 2026-06-19 14:37:50 on dist-test-slave-32hj"
server_key: "ba2e86ddefe9b0fb79ece617497ff03a"
server_key_iv: "a60aa1529aa6f9703f002ccae9c7ab56"
server_key_version: "encryptionkey@0"
I20260619 14:37:51.839797 7263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:37:51.857038 7263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:37:51.860525 7263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:37:51.860694 7263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:37:51.861294 7263 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:37:51.862107 7263 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:37:51.862112 6837 heartbeater.cc:503] Master 127.0.98.62:46665 was elected leader, sending a full tablet report...
I20260619 14:37:51.862155 7263 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:51.862207 7263 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:37:51.862236 7263 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:51.870715 7263 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33487
I20260619 14:37:51.870760 7480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33487 every 8 connection(s)
I20260619 14:37:51.871610 7263 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:37:51.872570 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 7263
I20260619 14:37:51.872766 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 6708
I20260619 14:37:51.879263 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:46221
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:46665
--webserver_port=46665
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:46665,127.0.98.61:39319
--tserver_master_addrs=127.0.98.62:46665,127.0.98.61:39319,127.0.98.60:39349 with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:37:51 dist-test-slave-32hj krb5kdc[6467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879871, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 19 14:37:51 dist-test-slave-32hj krb5kdc[6467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879871, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Jun 19 14:37:51 dist-test-slave-32hj krb5kdc[6467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879871, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 14:37:51.887497 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:51.878483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59857 (local address 127.0.98.61:39319)
0619 14:37:51.879060 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:51.879064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:51.879095 (+ 31us) server_negotiation.cc:407] Connection header received
0619 14:37:51.879138 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:51.879141 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:51.879187 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:51.879266 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:51.879797 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.880504 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.881630 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.881946 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.882013 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:51.884187 (+ 2174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:51.884191 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:51.884193 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:51.884218 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:51.885641 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:51.886092 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:51.886095 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:51.886097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:51.886175 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:51.886447 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:51.886450 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:51.886451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:51.886641 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:51.886720 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:51.887214 (+ 494us) server_negotiation.cc:299] Negotiation successful
0619 14:37:51.887328 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476}
I20260619 14:37:51.888124 7486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:51.873894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49439 (local address 127.0.98.60:39349)
0619 14:37:51.875434 (+ 1540us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:51.875439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:51.875454 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:37:51.875955 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:51.875958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:51.876010 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:51.876103 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:51.877253 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.878269 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.879969 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.880286 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.880393 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:51.883413 (+ 3020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:51.883417 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:51.883420 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:51.883447 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:51.885051 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:51.885662 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:51.885666 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:51.885668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:51.885755 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:51.886219 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:51.886222 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:51.886224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:51.886453 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:51.886546 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:51.887831 (+ 1285us) server_negotiation.cc:299] Negotiation successful
0619 14:37:51.887958 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1463,"thread_start_us":83,"threads_started":1}
I20260619 14:37:51.888828 7487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:51.873713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39319 (local address 127.0.98.1:59857)
0619 14:37:51.874661 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:51.874714 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:51.875741 (+ 1027us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:51.879296 (+ 3555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:51.879298 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:51.879505 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:51.879633 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:51.879641 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.880632 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.880635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:51.881504 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:51.881514 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.882152 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.882156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:51.882242 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:51.882682 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:51.882703 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:51.884041 (+ 1338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:51.885774 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:51.885777 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:51.885779 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:51.885990 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:51.886279 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:51.886281 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:51.886283 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:51.886361 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:51.886796 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:51.886799 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:51.886927 (+ 128us) client_negotiation.cc:769] Sending connection context
0619 14:37:51.887631 (+ 704us) client_negotiation.cc:240] Negotiation successful
0619 14:37:51.887769 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":762,"spinlock_wait_cycles":8192,"thread_start_us":116,"threads_started":1}
I20260619 14:37:51.888825 7488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:51.873803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39349 (local address 127.0.98.1:49439)
0619 14:37:51.874920 (+ 1117us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:51.874932 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:51.875746 (+ 814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:51.876245 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:51.876251 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:51.876585 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:51.877121 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:51.877134 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.878429 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.878433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:51.879835 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:51.879846 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.880570 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.880574 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:51.880805 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:51.881441 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:51.881464 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:51.883203 (+ 1739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:51.885181 (+ 1978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:51.885187 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:51.885204 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:51.885546 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:51.885856 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:51.885859 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:51.885861 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:51.886111 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:51.886601 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:51.886608 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:51.886930 (+ 322us) client_negotiation.cc:769] Sending connection context
0619 14:37:51.887143 (+ 213us) client_negotiation.cc:240] Negotiation successful
0619 14:37:51.887490 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1061,"mutex_wait_us":1,"spinlock_wait_cycles":54912,"thread_start_us":579,"threads_started":1}
I20260619 14:37:51.889633 6551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:51.874807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45515 (local address 127.0.98.62:46665)
0619 14:37:51.874922 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:51.874926 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:51.879907 (+ 4981us) server_negotiation.cc:407] Connection header received
0619 14:37:51.880059 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:51.880062 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:51.880110 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:51.880157 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:51.880691 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.881763 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.883101 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.883440 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.883518 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:51.886402 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:51.886406 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:51.886409 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:51.886433 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:51.887937 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:51.888455 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:51.888459 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:51.888461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:51.888538 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:51.888831 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:51.888834 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:51.888836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:51.889031 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:51.889097 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:51.889406 (+ 309us) server_negotiation.cc:299] Negotiation successful
0619 14:37:51.889503 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 14:37:51.890159 7481 heartbeater.cc:347] Connected to a master server at 127.0.98.60:39349
I20260619 14:37:51.890312 7485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:51.874705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46665 (local address 127.0.98.1:45515)
0619 14:37:51.879835 (+ 5130us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:51.879848 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:51.879964 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:51.880293 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:51.880296 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:51.880466 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:51.880571 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:51.880577 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.881879 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.881882 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:51.882973 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:51.882983 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:51.883666 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:51.883669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:51.883996 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:51.884738 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:51.884753 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:51.886273 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:51.888128 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:51.888131 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:51.888133 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:51.888352 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:51.888635 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:51.888637 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:51.888639 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:51.888734 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:51.889163 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:51.889166 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:51.889253 (+ 87us) client_negotiation.cc:769] Sending connection context
0619 14:37:51.889336 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 14:37:51.889450 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5068,"thread_start_us":74,"threads_started":1}
I20260619 14:37:51.890390 7481 heartbeater.cc:464] Registering TS with master...
I20260619 14:37:51.890671 7483 heartbeater.cc:347] Connected to a master server at 127.0.98.61:39319
I20260619 14:37:51.890745 7483 heartbeater.cc:464] Registering TS with master...
I20260619 14:37:51.890939 7482 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46665
I20260619 14:37:51.890933 7481 heartbeater.cc:511] Master 127.0.98.60:39349 requested a full tablet report, sending...
I20260619 14:37:51.890933 7483 heartbeater.cc:511] Master 127.0.98.61:39319 requested a full tablet report, sending...
I20260619 14:37:51.891016 7482 heartbeater.cc:464] Registering TS with master...
I20260619 14:37:51.891162 7482 heartbeater.cc:511] Master 127.0.98.62:46665 requested a full tablet report, sending...
I20260619 14:37:51.892243 6512 ts_manager.cc:194] Re-registered known tserver with Master: 57ffea6d72ae469491ce019f5e03a885 (127.0.98.1:33487)
I20260619 14:37:51.892438 7302 ts_manager.cc:194] Registered new tserver with Master: 57ffea6d72ae469491ce019f5e03a885 (127.0.98.1:33487)
I20260619 14:37:51.892437 7119 ts_manager.cc:194] Registered new tserver with Master: 57ffea6d72ae469491ce019f5e03a885 (127.0.98.1:33487)
I20260619 14:37:51.892963 6512 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45515
W20260619 14:37:51.990254 7489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:37:51.990494 7489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:37:51.990550 7489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:37:51.994424 7489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:37:51.994478 7489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:37:51.994556 7489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:37:51.999562 7489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:46221
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=46665
--tserver_master_addrs=127.0.98.62:46665,127.0.98.61:39319,127.0.98.60:39349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:37:52.000515 7489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:37:52.001516 7489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:37:52.010686 7495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:52.010746 7489 server_base.cc:1034] running on GCE node
W20260619 14:37:52.010686 7494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:52.010919 7497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:52.011351 7489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:37:52.011956 7489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:37:52.013125 7489 hybrid_clock.cc:648] HybridClock initialized: now 1781879872013110 us; error 24 us; skew 500 ppm
Jun 19 14:37:52 dist-test-slave-32hj krb5kdc[6467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879872, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:37:52.015614 7489 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260619 14:37:52.016433 7489 webserver.cc:460] Webserver started at http://127.0.98.2:46665/ using document root <none> and password file <none>
I20260619 14:37:52.016975 7489 fs_manager.cc:359] Metadata directory not provided
I20260619 14:37:52.017051 7489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:37:52.020525 7489 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:37:52.022389 7504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:52.023495 7489 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260619 14:37:52.023602 7489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-1/wal
uuid: "24dc9c1322cb4b5092de8d6a4c742d77"
format_stamp: "Formatted at 2026-06-19 14:37:50 on dist-test-slave-32hj"
server_key: "5ad151613ecbcf214f60f1f122e25a77"
server_key_iv: "fef86f629144ef2ab94a2bcfac6ad2b7"
server_key_version: "encryptionkey@0"
I20260619 14:37:52.024096 7489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:37:52.039916 7489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:37:52.042675 7489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:37:52.042843 7489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:37:52.043365 7489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:37:52.044135 7489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:37:52.044179 7489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:52.044227 7489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:37:52.044256 7489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:52.052354 7489 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46221
I20260619 14:37:52.052378 7617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46221 every 8 connection(s)
I20260619 14:37:52.053323 7489 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:37:52.055477 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 7489
I20260619 14:37:52.055642 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 6842
I20260619 14:37:52.062074 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:36403
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:46665
--webserver_port=40301
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:46665,127.0.98.61:39319
--tserver_master_addrs=127.0.98.62:46665,127.0.98.61:39319,127.0.98.60:39349 with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:37:52 dist-test-slave-32hj krb5kdc[6467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879872, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 19 14:37:52 dist-test-slave-32hj krb5kdc[6467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879872, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 19 14:37:52 dist-test-slave-32hj krb5kdc[6467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879872, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 14:37:52.070387 6551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:52.054959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38369 (local address 127.0.98.62:46665)
0619 14:37:52.055193 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:52.055197 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:52.056999 (+ 1802us) server_negotiation.cc:407] Connection header received
0619 14:37:52.058115 (+ 1116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:52.058118 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:52.058169 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:52.058211 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:52.059690 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:52.060546 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:52.062465 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:52.062782 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:52.062854 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:52.066229 (+ 3375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:52.066234 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:52.066236 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:52.066262 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:52.068231 (+ 1969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:52.069013 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:52.069017 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:52.069019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:52.069110 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:52.069429 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:52.069433 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:52.069435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:52.069649 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:52.069731 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:52.070079 (+ 348us) server_negotiation.cc:299] Negotiation successful
0619 14:37:52.070201 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158}
I20260619 14:37:52.071029 7486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:52.054958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51835 (local address 127.0.98.60:39349)
0619 14:37:52.055091 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:52.055096 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:52.056968 (+ 1872us) server_negotiation.cc:407] Connection header received
0619 14:37:52.058114 (+ 1146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:52.058118 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:52.058168 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:52.058216 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:52.059730 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:52.060535 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:52.062253 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:52.062575 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:52.062647 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:52.067304 (+ 4657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:52.067308 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:52.067311 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:52.067342 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:52.069426 (+ 2084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:52.069851 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:52.069855 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:52.069856 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:52.069935 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:52.070196 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:52.070200 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:52.070201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:52.070397 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:52.070470 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:52.070762 (+ 292us) server_negotiation.cc:299] Negotiation successful
0619 14:37:52.070891 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 14:37:52.071348 7622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:52.056489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39319 (local address 127.0.98.2:53231)
0619 14:37:52.057544 (+ 1055us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:52.057554 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:52.057886 (+ 332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:52.061341 (+ 3455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:52.061344 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:52.061526 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:52.061626 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:52.061632 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:52.062590 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:52.062592 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:52.063367 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:52.063377 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:52.063968 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:52.063972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:52.064058 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:52.064496 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:52.064510 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:52.066228 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:52.068144 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:52.068150 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:52.068166 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:52.068518 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:52.068828 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:52.068831 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:52.068833 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:52.069007 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:52.069496 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:52.069503 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:52.069802 (+ 299us) client_negotiation.cc:769] Sending connection context
0619 14:37:52.069998 (+ 196us) client_negotiation.cc:240] Negotiation successful
0619 14:37:52.070337 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1000,"thread_start_us":67,"threads_started":1}
I20260619 14:37:52.071506 7624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:52.055442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39349 (local address 127.0.98.2:51835)
0619 14:37:52.056821 (+ 1379us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:52.056876 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:52.057886 (+ 1010us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:52.058471 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:52.058476 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:52.058936 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:52.059531 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:52.059544 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:52.060692 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:52.060696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:52.062121 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:52.062131 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:52.062841 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:52.062844 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:52.063025 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:52.063708 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:52.063728 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:52.067172 (+ 3444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:52.069523 (+ 2351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:52.069526 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:52.069528 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:52.069731 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:52.070031 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:52.070033 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:52.070035 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:52.070107 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:52.070521 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:52.070523 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:52.070599 (+ 76us) client_negotiation.cc:769] Sending connection context
0619 14:37:52.071202 (+ 603us) client_negotiation.cc:240] Negotiation successful
0619 14:37:52.071325 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1150,"mutex_wait_us":99,"spinlock_wait_cycles":12800,"thread_start_us":132,"threads_started":1}
I20260619 14:37:52.072041 7623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:52.055442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46665 (local address 127.0.98.2:38369)
0619 14:37:52.056897 (+ 1455us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:52.056912 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:52.057887 (+ 975us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:52.058468 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:52.058475 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:52.058935 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:52.059529 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:52.059542 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:52.060791 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:52.060793 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:52.062122 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:52.062131 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:52.062999 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:52.063003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:52.063099 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:52.063801 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:52.063816 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:52.066056 (+ 2240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:52.068350 (+ 2294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:52.068353 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:52.068355 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:52.068894 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:52.069218 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:52.069221 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:52.069223 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:52.069311 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:52.069791 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:52.069794 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:52.069883 (+ 89us) client_negotiation.cc:769] Sending connection context
0619 14:37:52.070799 (+ 916us) client_negotiation.cc:240] Negotiation successful
0619 14:37:52.070938 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1362,"mutex_wait_us":156,"spinlock_wait_cycles":58112,"thread_start_us":622,"threads_started":1}
I20260619 14:37:52.072098 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:52.060924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53231 (local address 127.0.98.61:39319)
0619 14:37:52.061062 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:52.061067 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:52.061080 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:37:52.061120 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:52.061122 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:52.061166 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:52.061237 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:52.061756 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:52.062469 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:52.063472 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:52.063734 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:52.063845 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:52.066384 (+ 2539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:52.066388 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:52.066391 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:52.066417 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:52.068017 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:52.068627 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:52.068630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:52.068632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:52.068708 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:52.069114 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:52.069118 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:52.069120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:52.069359 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:52.069444 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:52.071852 (+ 2408us) server_negotiation.cc:299] Negotiation successful
0619 14:37:52.071962 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 14:37:52.072849 7618 heartbeater.cc:347] Connected to a master server at 127.0.98.60:39349
I20260619 14:37:52.072922 7620 heartbeater.cc:347] Connected to a master server at 127.0.98.61:39319
I20260619 14:37:52.073078 7618 heartbeater.cc:464] Registering TS with master...
I20260619 14:37:52.073107 7619 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46665
I20260619 14:37:52.073078 7620 heartbeater.cc:464] Registering TS with master...
I20260619 14:37:52.073169 7619 heartbeater.cc:464] Registering TS with master...
I20260619 14:37:52.073585 7618 heartbeater.cc:511] Master 127.0.98.60:39349 requested a full tablet report, sending...
I20260619 14:37:52.073585 7620 heartbeater.cc:511] Master 127.0.98.61:39319 requested a full tablet report, sending...
I20260619 14:37:52.073724 7619 heartbeater.cc:511] Master 127.0.98.62:46665 requested a full tablet report, sending...
I20260619 14:37:52.074661 7302 ts_manager.cc:194] Registered new tserver with Master: 24dc9c1322cb4b5092de8d6a4c742d77 (127.0.98.2:46221)
I20260619 14:37:52.074798 7119 ts_manager.cc:194] Registered new tserver with Master: 24dc9c1322cb4b5092de8d6a4c742d77 (127.0.98.2:46221)
I20260619 14:37:52.075148 6512 ts_manager.cc:194] Re-registered known tserver with Master: 24dc9c1322cb4b5092de8d6a4c742d77 (127.0.98.2:46221)
I20260619 14:37:52.075722 6512 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38369
W20260619 14:37:52.173775 7625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:37:52.174048 7625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:37:52.174104 7625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:37:52.178319 7625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:37:52.178378 7625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:37:52.178457 7625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:37:52.183638 7625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:36403
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40301
--tserver_master_addrs=127.0.98.62:46665,127.0.98.61:39319,127.0.98.60:39349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:37:52.184628 7625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:37:52.185591 7625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:37:52.194479 7631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:52.194486 7630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:52.194514 7633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:52.194749 7625 server_base.cc:1034] running on GCE node
I20260619 14:37:52.195122 7625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:37:52.195695 7625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:37:52.196874 7625 hybrid_clock.cc:648] HybridClock initialized: now 1781879872196861 us; error 29 us; skew 500 ppm
Jun 19 14:37:52 dist-test-slave-32hj krb5kdc[6467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879872, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:37:52.199393 7625 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260619 14:37:52.200160 7625 webserver.cc:460] Webserver started at http://127.0.98.3:40301/ using document root <none> and password file <none>
I20260619 14:37:52.200649 7625 fs_manager.cc:359] Metadata directory not provided
I20260619 14:37:52.200713 7625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:37:52.204293 7625 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:37:52.206396 7640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:52.207755 7625 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:37:52.207875 7625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-2/wal
uuid: "77032b2db22744feb997087f373b1209"
format_stamp: "Formatted at 2026-06-19 14:37:51 on dist-test-slave-32hj"
server_key: "9c09b8adc44e52152472901bd7c47ede"
server_key_iv: "df202399ce48ccc156a33c5030bb347f"
server_key_version: "encryptionkey@0"
I20260619 14:37:52.208406 7625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:37:52.223184 7625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:37:52.226085 7625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:37:52.226248 7625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:37:52.226825 7625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:37:52.227707 7625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:37:52.227756 7625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:52.227806 7625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:37:52.227834 7625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:52.236692 7625 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36403
I20260619 14:37:52.236760 7753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36403 every 8 connection(s)
I20260619 14:37:52.237708 7625 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:37:52.238447 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 7625
Jun 19 14:37:52 dist-test-slave-32hj krb5kdc[6467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879872, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 14:37:52.250694 7771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:52.242056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39349 (local address 127.0.0.1:57592)
0619 14:37:52.245856 (+ 3800us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:52.245869 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:52.245950 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:52.246305 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:52.246308 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:52.246334 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:37:52.246409 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:52.246415 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:52.247668 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:52.247672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:52.248828 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:52.248841 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:52.250266 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:52.250270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:52.250367 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:52.250391 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 14:37:52.250455 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 14:37:52.250535 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3739,"thread_start_us":60,"threads_started":1}
I20260619 14:37:52.251119 7772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:52.242319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57592 (local address 127.0.98.60:39349)
0619 14:37:52.245724 (+ 3405us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:52.245736 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:52.246025 (+ 289us) server_negotiation.cc:407] Connection header received
0619 14:37:52.246065 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:52.246068 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:52.246123 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:52.246216 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:37:52.246519 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:52.247207 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:52.249320 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:52.249954 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:52.250052 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:52.250202 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:52.250914 (+ 712us) server_negotiation.cc:299] Negotiation successful
0619 14:37:52.250993 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3340,"thread_start_us":82,"threads_started":1}
I20260619 14:37:52.251668 7312 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
Jun 19 14:37:52 dist-test-slave-32hj krb5kdc[6467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879872, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260619 14:37:52.252231 7344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e41b19ce379e45198aeff6fd3814bb43. Latest consensus state: current_term: 1 leader_uuid: "e41b19ce379e45198aeff6fd3814bb43" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e41b19ce379e45198aeff6fd3814bb43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46665 } } peers { permanent_uuid: "6731bc26510d493c99ecc1aba97fd05b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39319 } attrs { promote: false } } peers { permanent_uuid: "827bf26a4cd440e59c8db36bbc95d991" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39349 } attrs { promote: true } } }
I20260619 14:37:52.252408 7344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991 [sys.catalog]: This master's current role is: LEARNER
I20260619 14:37:52.253619 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:52.240558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47454 (local address 127.0.98.61:39319)
0619 14:37:52.240664 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:52.240668 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:52.241645 (+ 977us) server_negotiation.cc:407] Connection header received
0619 14:37:52.241691 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:52.241694 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:52.241740 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:52.241822 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:52.242877 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:52.243649 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:52.244831 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:52.245158 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:52.245241 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:52.247763 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:52.247767 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:52.247769 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:52.247795 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:52.250291 (+ 2496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:52.252212 (+ 1921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:52.252215 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:52.252218 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:52.252305 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:52.252641 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:52.252645 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:52.252646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:52.252895 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:52.253004 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:52.253355 (+ 351us) server_negotiation.cc:299] Negotiation successful
0619 14:37:52.253469 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 14:37:52.255470 6551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:52.240143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48368 (local address 127.0.98.62:46665)
0619 14:37:52.240266 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:52.240270 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:52.241119 (+ 849us) server_negotiation.cc:407] Connection header received
0619 14:37:52.242628 (+ 1509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:52.242632 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:52.242677 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:52.242725 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:52.243599 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:52.244374 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:52.245440 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:52.246183 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:52.246245 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:52.248665 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:52.248669 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:52.248671 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:52.248694 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:52.251593 (+ 2899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:52.254258 (+ 2665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:52.254262 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:52.254264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:52.254346 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:52.254620 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:52.254623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:52.254624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:52.254795 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:52.254907 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:52.255226 (+ 319us) server_negotiation.cc:299] Negotiation successful
0619 14:37:52.255330 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 14:37:52.256502 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:52.240544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57580 (local address 127.0.98.60:39349)
0619 14:37:52.241859 (+ 1315us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:52.241863 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:52.241875 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:37:52.241912 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:52.241915 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:52.241956 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:52.242030 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:52.242843 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:52.243640 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:52.244748 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:52.245099 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:52.245185 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:52.247352 (+ 2167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:52.247356 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:52.247359 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:52.247385 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:52.253786 (+ 6401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:52.254877 (+ 1091us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:52.254880 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:52.254882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:52.254966 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:52.255239 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:52.255242 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:52.255244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:52.255459 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:52.256179 (+ 720us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:52.256242 (+ 63us) server_negotiation.cc:299] Negotiation successful
0619 14:37:52.256357 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1250,"thread_start_us":73,"threads_started":1}
I20260619 14:37:52.257038 7486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:52.239631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37079 (local address 127.0.98.60:39349)
0619 14:37:52.239783 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:52.239788 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:52.241515 (+ 1727us) server_negotiation.cc:407] Connection header received
0619 14:37:52.242617 (+ 1102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:52.242620 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:52.242665 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:52.242728 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:52.248083 (+ 5355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:52.248770 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:52.249821 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:52.250172 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:52.250243 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:52.252710 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:52.252714 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:52.252747 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:52.252771 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:52.254449 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:52.255382 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:52.255385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:52.255387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:52.255451 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:52.255852 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:52.255856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:52.255858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:52.256051 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:52.256117 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:52.256795 (+ 678us) server_negotiation.cc:299] Negotiation successful
0619 14:37:52.256905 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 14:37:52.258523 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:52.239956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39349 (local address 127.0.98.3:37079)
0619 14:37:52.241377 (+ 1421us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:52.241431 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:52.242402 (+ 971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:52.245610 (+ 3208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:52.245613 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:52.245784 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:52.245908 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:52.245915 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:52.248882 (+ 2967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:52.248886 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:52.249696 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:52.249707 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:52.250480 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:52.250483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:52.250570 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:52.251001 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:52.251014 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:52.252587 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:52.254943 (+ 2356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:52.254949 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:52.254966 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:52.255284 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:52.255558 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:52.255561 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:52.255563 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:52.255732 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:52.256195 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:52.256201 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:52.256500 (+ 299us) client_negotiation.cc:769] Sending connection context
0619 14:37:52.257173 (+ 673us) client_negotiation.cc:240] Negotiation successful
0619 14:37:52.257506 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1009,"thread_start_us":620,"threads_started":1}
I20260619 14:37:52.258637 7767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:52.240836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43031 (local address 127.0.98.62:46665)
0619 14:37:52.241238 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:52.241242 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:52.242934 (+ 1692us) server_negotiation.cc:407] Connection header received
0619 14:37:52.242967 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:52.242970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:52.243007 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:52.243141 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:52.244768 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:52.245486 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:52.247556 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:52.247870 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:52.247956 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:52.250992 (+ 3036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:52.250996 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:52.250999 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:52.251021 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:52.252804 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:52.257279 (+ 4475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:52.257283 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:52.257284 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:52.257369 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:52.257654 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:52.257657 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:52.257659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:52.257854 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:52.257932 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:52.258385 (+ 453us) server_negotiation.cc:299] Negotiation successful
0619 14:37:52.258492 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":68,"threads_started":1}
I20260619 14:37:52.259174 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:52.239956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46665 (local address 127.0.98.3:43031)
0619 14:37:52.242760 (+ 2804us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:52.242771 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:52.242865 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:52.243405 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:52.243414 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:52.243902 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:52.244597 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:52.244616 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:52.246052 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:52.246055 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:52.247424 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:52.247435 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:52.248154 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:52.248158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:52.248346 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:52.249036 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:52.249055 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:52.250817 (+ 1762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:52.256875 (+ 6058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:52.256878 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:52.256880 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:52.257173 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:52.257470 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:52.257473 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:52.257474 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:52.257557 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:52.258018 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:52.258021 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:52.258120 (+ 99us) client_negotiation.cc:769] Sending connection context
0619 14:37:52.258209 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 14:37:52.259008 (+ 799us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2716,"mutex_wait_us":23,"spinlock_wait_cycles":41728,"thread_start_us":149,"threads_started":1}
I20260619 14:37:52.259687 7754 heartbeater.cc:347] Connected to a master server at 127.0.98.60:39349
I20260619 14:37:52.259925 7754 heartbeater.cc:464] Registering TS with master...
Jun 19 14:37:52 dist-test-slave-32hj krb5kdc[6467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879872, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 14:37:52.259969 7755 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46665
I20260619 14:37:52.260025 7755 heartbeater.cc:464] Registering TS with master...
I20260619 14:37:52.260486 7755 heartbeater.cc:511] Master 127.0.98.62:46665 requested a full tablet report, sending...
I20260619 14:37:52.260486 7754 heartbeater.cc:511] Master 127.0.98.60:39349 requested a full tablet report, sending...
I20260619 14:37:52.258463 6512 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48368:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 14:37:52.261610 6508 ts_manager.cc:194] Re-registered known tserver with Master: 77032b2db22744feb997087f373b1209 (127.0.98.3:36403)
I20260619 14:37:52.262013 7302 ts_manager.cc:194] Registered new tserver with Master: 77032b2db22744feb997087f373b1209 (127.0.98.3:36403)
W20260619 14:37:52.262236 6512 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:37:52.262281 6508 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43031
I20260619 14:37:52.264892 7770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:52.241515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55197 (local address 127.0.98.61:39319)
0619 14:37:52.241740 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:52.241744 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:52.253383 (+ 11639us) server_negotiation.cc:407] Connection header received
0619 14:37:52.253422 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:52.253424 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:52.253470 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:52.253544 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:52.254065 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:52.254792 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:52.255893 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:52.256329 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:52.256414 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:52.261476 (+ 5062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:52.261479 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:52.261482 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:52.261503 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:52.263263 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:52.263729 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:52.263732 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:52.263734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:52.263831 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:52.264102 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:52.264105 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:52.264106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:52.264310 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:52.264398 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:52.264599 (+ 201us) server_negotiation.cc:299] Negotiation successful
0619 14:37:52.264712 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":71,"threads_started":1}
I20260619 14:37:52.264882 7769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:52.241289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39319 (local address 127.0.98.3:55197)
0619 14:37:52.253190 (+ 11901us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:52.253205 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:52.253296 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:52.253658 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:52.253661 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:52.253831 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:52.253947 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:52.253955 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:52.254904 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:52.254907 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:52.255773 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:52.255784 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:52.256534 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:52.256538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:52.256623 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:52.258167 (+ 1544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:52.258179 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:52.261336 (+ 3157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:52.263381 (+ 2045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:52.263384 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:52.263386 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:52.263610 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:52.263919 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:52.263921 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:52.263922 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:52.264023 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:52.264394 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:52.264396 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:52.264460 (+ 64us) client_negotiation.cc:769] Sending connection context
0619 14:37:52.264529 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 14:37:52.264659 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11834,"thread_start_us":73,"threads_started":1}
I20260619 14:37:52.264994 7071 sys_catalog.cc:455] T 00000000000000000000000000000000 P e41b19ce379e45198aeff6fd3814bb43 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e41b19ce379e45198aeff6fd3814bb43" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e41b19ce379e45198aeff6fd3814bb43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46665 } } peers { permanent_uuid: "6731bc26510d493c99ecc1aba97fd05b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39319 } attrs { promote: false } } peers { permanent_uuid: "827bf26a4cd440e59c8db36bbc95d991" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39349 } attrs { promote: true } } }
I20260619 14:37:52.265100 7071 sys_catalog.cc:458] T 00000000000000000000000000000000 P e41b19ce379e45198aeff6fd3814bb43 [sys.catalog]: This master's current role is: LEADER
I20260619 14:37:52.265841 7756 heartbeater.cc:347] Connected to a master server at 127.0.98.61:39319
I20260619 14:37:52.265924 7756 heartbeater.cc:464] Registering TS with master...
I20260619 14:37:52.266093 7756 heartbeater.cc:511] Master 127.0.98.61:39319 requested a full tablet report, sending...
I20260619 14:37:52.266737 7119 ts_manager.cc:194] Registered new tserver with Master: 77032b2db22744feb997087f373b1209 (127.0.98.3:36403)
I20260619 14:37:52.268122 7774 raft_consensus.cc:1050] T 00000000000000000000000000000000 P e41b19ce379e45198aeff6fd3814bb43: attempting to promote NON_VOTER 827bf26a4cd440e59c8db36bbc95d991 to VOTER
I20260619 14:37:52.268450 7774 consensus_queue.cc:237] T 00000000000000000000000000000000 P e41b19ce379e45198aeff6fd3814bb43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e41b19ce379e45198aeff6fd3814bb43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46665 } } peers { permanent_uuid: "6731bc26510d493c99ecc1aba97fd05b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39319 } attrs { promote: false } } peers { permanent_uuid: "827bf26a4cd440e59c8db36bbc95d991" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39349 } attrs { promote: false } }
I20260619 14:37:52.269446 7312 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991 [term 1 LEARNER]: Refusing update from remote peer e41b19ce379e45198aeff6fd3814bb43: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260619 14:37:52.269507 7129 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b [term 1 FOLLOWER]: Refusing update from remote peer e41b19ce379e45198aeff6fd3814bb43: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260619 14:37:52.269797 7071 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e41b19ce379e45198aeff6fd3814bb43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "827bf26a4cd440e59c8db36bbc95d991" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39349 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260619 14:37:52.269934 7779 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e41b19ce379e45198aeff6fd3814bb43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6731bc26510d493c99ecc1aba97fd05b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260619 14:37:52.271513 7774 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e41b19ce379e45198aeff6fd3814bb43 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 827bf26a4cd440e59c8db36bbc95d991 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e41b19ce379e45198aeff6fd3814bb43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46665 } } peers { permanent_uuid: "6731bc26510d493c99ecc1aba97fd05b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39319 } attrs { promote: false } } peers { permanent_uuid: "827bf26a4cd440e59c8db36bbc95d991" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39349 } attrs { promote: false } } }
I20260619 14:37:52.271993 7312 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 827bf26a4cd440e59c8db36bbc95d991 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e41b19ce379e45198aeff6fd3814bb43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46665 } } peers { permanent_uuid: "6731bc26510d493c99ecc1aba97fd05b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39319 } attrs { promote: false } } peers { permanent_uuid: "827bf26a4cd440e59c8db36bbc95d991" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39349 } attrs { promote: false } } }
I20260619 14:37:52.273872 7344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e41b19ce379e45198aeff6fd3814bb43" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e41b19ce379e45198aeff6fd3814bb43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46665 } } peers { permanent_uuid: "6731bc26510d493c99ecc1aba97fd05b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39319 } attrs { promote: false } } peers { permanent_uuid: "827bf26a4cd440e59c8db36bbc95d991" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39349 } attrs { promote: false } } }
I20260619 14:37:52.273976 7344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:37:52.274356 7779 sys_catalog.cc:455] T 00000000000000000000000000000000 P e41b19ce379e45198aeff6fd3814bb43 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e41b19ce379e45198aeff6fd3814bb43" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e41b19ce379e45198aeff6fd3814bb43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46665 } } peers { permanent_uuid: "6731bc26510d493c99ecc1aba97fd05b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39319 } attrs { promote: false } } peers { permanent_uuid: "827bf26a4cd440e59c8db36bbc95d991" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39349 } attrs { promote: false } } }
I20260619 14:37:52.274444 7779 sys_catalog.cc:458] T 00000000000000000000000000000000 P e41b19ce379e45198aeff6fd3814bb43 [sys.catalog]: This master's current role is: LEADER
I20260619 14:37:52.274518 7129 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 827bf26a4cd440e59c8db36bbc95d991 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e41b19ce379e45198aeff6fd3814bb43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46665 } } peers { permanent_uuid: "6731bc26510d493c99ecc1aba97fd05b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39319 } attrs { promote: false } } peers { permanent_uuid: "827bf26a4cd440e59c8db36bbc95d991" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39349 } attrs { promote: false } } }
I20260619 14:37:52.274957 7774 sys_catalog.cc:455] T 00000000000000000000000000000000 P e41b19ce379e45198aeff6fd3814bb43 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e41b19ce379e45198aeff6fd3814bb43" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e41b19ce379e45198aeff6fd3814bb43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46665 } } peers { permanent_uuid: "6731bc26510d493c99ecc1aba97fd05b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39319 } attrs { promote: false } } peers { permanent_uuid: "827bf26a4cd440e59c8db36bbc95d991" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39349 } attrs { promote: false } } }
I20260619 14:37:52.275048 7774 sys_catalog.cc:458] T 00000000000000000000000000000000 P e41b19ce379e45198aeff6fd3814bb43 [sys.catalog]: This master's current role is: LEADER
I20260619 14:37:52.274998 7779 sys_catalog.cc:455] T 00000000000000000000000000000000 P e41b19ce379e45198aeff6fd3814bb43 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e41b19ce379e45198aeff6fd3814bb43" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e41b19ce379e45198aeff6fd3814bb43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46665 } } peers { permanent_uuid: "6731bc26510d493c99ecc1aba97fd05b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39319 } attrs { promote: false } } peers { permanent_uuid: "827bf26a4cd440e59c8db36bbc95d991" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39349 } attrs { promote: false } } }
I20260619 14:37:52.275089 7779 sys_catalog.cc:458] T 00000000000000000000000000000000 P e41b19ce379e45198aeff6fd3814bb43 [sys.catalog]: This master's current role is: LEADER
I20260619 14:37:52.276144 7783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e41b19ce379e45198aeff6fd3814bb43" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e41b19ce379e45198aeff6fd3814bb43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46665 } } peers { permanent_uuid: "6731bc26510d493c99ecc1aba97fd05b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39319 } attrs { promote: false } } peers { permanent_uuid: "827bf26a4cd440e59c8db36bbc95d991" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39349 } attrs { promote: false } } }
I20260619 14:37:52.276268 7783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:37:52.283694 7771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:52.279155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36403 (local address 127.0.0.1:34938)
0619 14:37:52.279527 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:52.279553 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:52.279601 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:52.280230 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:52.280232 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:52.280248 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:37:52.280330 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:52.280335 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:52.281471 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:52.281474 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:52.282256 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:52.282264 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:52.283326 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:52.283330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:52.283407 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:52.283432 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 14:37:52.283486 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 14:37:52.283556 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":100,"threads_started":1}
I20260619 14:37:52.283749 7786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:52.279317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34938 (local address 127.0.98.3:36403)
0619 14:37:52.279768 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:52.279777 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:52.279797 (+ 20us) server_negotiation.cc:407] Connection header received
0619 14:37:52.279867 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:52.279872 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:52.280101 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:52.280263 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:37:52.280441 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:52.281382 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:52.282386 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:52.283200 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:52.283311 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:52.283469 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:52.283518 (+ 49us) server_negotiation.cc:299] Negotiation successful
0619 14:37:52.283607 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":113,"threads_started":1}
I20260619 14:37:52.285171 7784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:52.278940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33487 (local address 127.0.0.1:43668)
0619 14:37:52.279446 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:52.279460 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:52.279582 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:52.280515 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:52.280518 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:52.280536 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:37:52.280626 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:52.280633 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:52.282138 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:52.282141 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:52.283326 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:52.283336 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:52.284807 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:52.284811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:52.284898 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:52.284921 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 14:37:52.284988 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 14:37:52.285052 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"mutex_wait_us":27}
I20260619 14:37:52.285336 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:52.279325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49850 (local address 127.0.98.2:46221)
0619 14:37:52.280545 (+ 1220us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:52.280557 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:52.280578 (+ 21us) server_negotiation.cc:407] Connection header received
0619 14:37:52.280648 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:52.280653 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:52.280893 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:52.281035 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:37:52.281219 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:52.282269 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:52.283482 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:52.284785 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:52.284887 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:52.285022 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:52.285113 (+ 91us) server_negotiation.cc:299] Negotiation successful
0619 14:37:52.285203 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1115,"thread_start_us":105,"threads_started":1}
I20260619 14:37:52.285327 7785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:52.279155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46221 (local address 127.0.0.1:49850)
0619 14:37:52.279744 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:52.279755 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:52.279823 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:52.281032 (+ 1209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:52.281035 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:52.281054 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:37:52.281129 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:52.281134 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:52.282387 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:52.282391 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:52.283356 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:52.283364 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:52.284900 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:52.284903 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:52.284989 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:52.285010 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 14:37:52.285085 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 14:37:52.285154 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"mutex_wait_us":39,"thread_start_us":88,"threads_started":1}
I20260619 14:37:52.286136 7788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:52.279752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43668 (local address 127.0.98.1:33487)
0619 14:37:52.280061 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:52.280071 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:52.280091 (+ 20us) server_negotiation.cc:407] Connection header received
0619 14:37:52.280160 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:52.280165 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:52.280372 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:52.280780 (+ 408us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:37:52.280913 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:52.282023 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:52.283451 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:52.284357 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:52.284457 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:52.284596 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:52.285904 (+ 1308us) server_negotiation.cc:299] Negotiation successful
0619 14:37:52.286006 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":108,"threads_started":1}
I20260619 14:37:52.287088 7688 tablet_service.cc:1467] Processing CreateTablet for tablet b9f7fbb6d1b04ae98dad245c92e55cec (DEFAULT_TABLE table=test-table [id=12b304f3d9234f65b032b3e927bed4b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:37:52.288031 7688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9f7fbb6d1b04ae98dad245c92e55cec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:52.288154 7552 tablet_service.cc:1467] Processing CreateTablet for tablet b9f7fbb6d1b04ae98dad245c92e55cec (DEFAULT_TABLE table=test-table [id=12b304f3d9234f65b032b3e927bed4b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:37:52.289245 7552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9f7fbb6d1b04ae98dad245c92e55cec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:52.289057 7415 tablet_service.cc:1467] Processing CreateTablet for tablet b9f7fbb6d1b04ae98dad245c92e55cec (DEFAULT_TABLE table=test-table [id=12b304f3d9234f65b032b3e927bed4b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:37:52.290380 7415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9f7fbb6d1b04ae98dad245c92e55cec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:52.295307 7789 tablet_bootstrap.cc:492] T b9f7fbb6d1b04ae98dad245c92e55cec P 77032b2db22744feb997087f373b1209: Bootstrap starting.
I20260619 14:37:52.297446 7789 tablet_bootstrap.cc:654] T b9f7fbb6d1b04ae98dad245c92e55cec P 77032b2db22744feb997087f373b1209: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:52.298870 7789 log.cc:824] T b9f7fbb6d1b04ae98dad245c92e55cec P 77032b2db22744feb997087f373b1209: Log is configured to *not* fsync() on all Append() calls
I20260619 14:37:52.298951 7791 tablet_bootstrap.cc:492] T b9f7fbb6d1b04ae98dad245c92e55cec P 57ffea6d72ae469491ce019f5e03a885: Bootstrap starting.
I20260619 14:37:52.300837 7790 tablet_bootstrap.cc:492] T b9f7fbb6d1b04ae98dad245c92e55cec P 24dc9c1322cb4b5092de8d6a4c742d77: Bootstrap starting.
I20260619 14:37:52.301122 7791 tablet_bootstrap.cc:654] T b9f7fbb6d1b04ae98dad245c92e55cec P 57ffea6d72ae469491ce019f5e03a885: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:52.301373 7789 tablet_bootstrap.cc:492] T b9f7fbb6d1b04ae98dad245c92e55cec P 77032b2db22744feb997087f373b1209: No bootstrap required, opened a new log
I20260619 14:37:52.301563 7789 ts_tablet_manager.cc:1397] T b9f7fbb6d1b04ae98dad245c92e55cec P 77032b2db22744feb997087f373b1209: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 14:37:52.301939 7791 log.cc:824] T b9f7fbb6d1b04ae98dad245c92e55cec P 57ffea6d72ae469491ce019f5e03a885: Log is configured to *not* fsync() on all Append() calls
I20260619 14:37:52.302685 7790 tablet_bootstrap.cc:654] T b9f7fbb6d1b04ae98dad245c92e55cec P 24dc9c1322cb4b5092de8d6a4c742d77: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:52.303308 7790 log.cc:824] T b9f7fbb6d1b04ae98dad245c92e55cec P 24dc9c1322cb4b5092de8d6a4c742d77: Log is configured to *not* fsync() on all Append() calls
I20260619 14:37:52.303906 7791 tablet_bootstrap.cc:492] T b9f7fbb6d1b04ae98dad245c92e55cec P 57ffea6d72ae469491ce019f5e03a885: No bootstrap required, opened a new log
I20260619 14:37:52.304096 7791 ts_tablet_manager.cc:1397] T b9f7fbb6d1b04ae98dad245c92e55cec P 57ffea6d72ae469491ce019f5e03a885: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:37:52.305258 7789 raft_consensus.cc:348] T b9f7fbb6d1b04ae98dad245c92e55cec P 77032b2db22744feb997087f373b1209 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ffea6d72ae469491ce019f5e03a885" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33487 } } peers { permanent_uuid: "77032b2db22744feb997087f373b1209" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36403 } } peers { permanent_uuid: "24dc9c1322cb4b5092de8d6a4c742d77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46221 } }
I20260619 14:37:52.305501 7790 tablet_bootstrap.cc:492] T b9f7fbb6d1b04ae98dad245c92e55cec P 24dc9c1322cb4b5092de8d6a4c742d77: No bootstrap required, opened a new log
I20260619 14:37:52.305532 7789 raft_consensus.cc:374] T b9f7fbb6d1b04ae98dad245c92e55cec P 77032b2db22744feb997087f373b1209 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:52.305583 7789 raft_consensus.cc:729] T b9f7fbb6d1b04ae98dad245c92e55cec P 77032b2db22744feb997087f373b1209 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77032b2db22744feb997087f373b1209, State: Initialized, Role: FOLLOWER
I20260619 14:37:52.305660 7790 ts_tablet_manager.cc:1397] T b9f7fbb6d1b04ae98dad245c92e55cec P 24dc9c1322cb4b5092de8d6a4c742d77: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 14:37:52.306041 7789 consensus_queue.cc:260] T b9f7fbb6d1b04ae98dad245c92e55cec P 77032b2db22744feb997087f373b1209 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ffea6d72ae469491ce019f5e03a885" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33487 } } peers { permanent_uuid: "77032b2db22744feb997087f373b1209" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36403 } } peers { permanent_uuid: "24dc9c1322cb4b5092de8d6a4c742d77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46221 } }
I20260619 14:37:52.307567 7789 ts_tablet_manager.cc:1428] T b9f7fbb6d1b04ae98dad245c92e55cec P 77032b2db22744feb997087f373b1209: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 14:37:52.308027 7755 heartbeater.cc:503] Master 127.0.98.62:46665 was elected leader, sending a full tablet report...
I20260619 14:37:52.308094 7791 raft_consensus.cc:348] T b9f7fbb6d1b04ae98dad245c92e55cec P 57ffea6d72ae469491ce019f5e03a885 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ffea6d72ae469491ce019f5e03a885" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33487 } } peers { permanent_uuid: "77032b2db22744feb997087f373b1209" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36403 } } peers { permanent_uuid: "24dc9c1322cb4b5092de8d6a4c742d77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46221 } }
I20260619 14:37:52.308351 7791 raft_consensus.cc:374] T b9f7fbb6d1b04ae98dad245c92e55cec P 57ffea6d72ae469491ce019f5e03a885 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:52.308413 7791 raft_consensus.cc:729] T b9f7fbb6d1b04ae98dad245c92e55cec P 57ffea6d72ae469491ce019f5e03a885 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57ffea6d72ae469491ce019f5e03a885, State: Initialized, Role: FOLLOWER
I20260619 14:37:52.308868 7791 consensus_queue.cc:260] T b9f7fbb6d1b04ae98dad245c92e55cec P 57ffea6d72ae469491ce019f5e03a885 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ffea6d72ae469491ce019f5e03a885" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33487 } } peers { permanent_uuid: "77032b2db22744feb997087f373b1209" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36403 } } peers { permanent_uuid: "24dc9c1322cb4b5092de8d6a4c742d77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46221 } }
I20260619 14:37:52.309805 7791 ts_tablet_manager.cc:1428] T b9f7fbb6d1b04ae98dad245c92e55cec P 57ffea6d72ae469491ce019f5e03a885: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 14:37:52.310273 7790 raft_consensus.cc:348] T b9f7fbb6d1b04ae98dad245c92e55cec P 24dc9c1322cb4b5092de8d6a4c742d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ffea6d72ae469491ce019f5e03a885" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33487 } } peers { permanent_uuid: "77032b2db22744feb997087f373b1209" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36403 } } peers { permanent_uuid: "24dc9c1322cb4b5092de8d6a4c742d77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46221 } }
I20260619 14:37:52.310510 7790 raft_consensus.cc:374] T b9f7fbb6d1b04ae98dad245c92e55cec P 24dc9c1322cb4b5092de8d6a4c742d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:52.310565 7790 raft_consensus.cc:729] T b9f7fbb6d1b04ae98dad245c92e55cec P 24dc9c1322cb4b5092de8d6a4c742d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24dc9c1322cb4b5092de8d6a4c742d77, State: Initialized, Role: FOLLOWER
I20260619 14:37:52.311112 7482 heartbeater.cc:503] Master 127.0.98.62:46665 was elected leader, sending a full tablet report...
I20260619 14:37:52.311632 7790 consensus_queue.cc:260] T b9f7fbb6d1b04ae98dad245c92e55cec P 24dc9c1322cb4b5092de8d6a4c742d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ffea6d72ae469491ce019f5e03a885" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33487 } } peers { permanent_uuid: "77032b2db22744feb997087f373b1209" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36403 } } peers { permanent_uuid: "24dc9c1322cb4b5092de8d6a4c742d77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46221 } }
I20260619 14:37:52.312559 7790 ts_tablet_manager.cc:1428] T b9f7fbb6d1b04ae98dad245c92e55cec P 24dc9c1322cb4b5092de8d6a4c742d77: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260619 14:37:52.312814 7619 heartbeater.cc:503] Master 127.0.98.62:46665 was elected leader, sending a full tablet report...
W20260619 14:37:52.373495 7484 tablet.cc:2369] T b9f7fbb6d1b04ae98dad245c92e55cec P 57ffea6d72ae469491ce019f5e03a885: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:37:52.375943 7796 raft_consensus.cc:482] T b9f7fbb6d1b04ae98dad245c92e55cec P 57ffea6d72ae469491ce019f5e03a885 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:37:52.376119 7796 raft_consensus.cc:504] T b9f7fbb6d1b04ae98dad245c92e55cec P 57ffea6d72ae469491ce019f5e03a885 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ffea6d72ae469491ce019f5e03a885" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33487 } } peers { permanent_uuid: "77032b2db22744feb997087f373b1209" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36403 } } peers { permanent_uuid: "24dc9c1322cb4b5092de8d6a4c742d77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46221 } }
I20260619 14:37:52.377296 7796 leader_election.cc:288] T b9f7fbb6d1b04ae98dad245c92e55cec P 57ffea6d72ae469491ce019f5e03a885 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77032b2db22744feb997087f373b1209 (127.0.98.3:36403), 24dc9c1322cb4b5092de8d6a4c742d77 (127.0.98.2:46221)
I20260619 14:37:52.381028 7786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:52.377787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49731 (local address 127.0.98.3:36403)
0619 14:37:52.377902 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:52.377907 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:52.377972 (+ 65us) server_negotiation.cc:407] Connection header received
0619 14:37:52.378028 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:52.378031 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:52.378089 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:52.378196 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:37:52.378470 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:52.379067 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:52.380132 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:52.380515 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:52.380608 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:52.380669 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:52.380852 (+ 183us) server_negotiation.cc:299] Negotiation successful
0619 14:37:52.380924 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 14:37:52.381043 7485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:52.377537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36403 (local address 127.0.98.1:49731)
0619 14:37:52.377658 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:52.377675 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:52.377760 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:52.378218 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:52.378221 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:52.378243 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:37:52.378376 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:52.378384 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:52.379165 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:52.379168 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:52.380020 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:52.380029 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:52.380626 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:52.380629 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:52.380737 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:52.380760 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 14:37:52.380827 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 14:37:52.380899 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260619 14:37:52.381626 7488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:52.377550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46221 (local address 127.0.98.1:40175)
0619 14:37:52.377954 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:52.377970 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:52.378050 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:52.378327 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:52.378330 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:52.378349 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:37:52.378453 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:52.378460 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:52.379254 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:52.379258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:52.380475 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:52.380484 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:52.381253 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:52.381257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:52.381362 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:52.381385 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 14:37:52.381453 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 14:37:52.381514 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":231}
I20260619 14:37:52.381656 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:52.377650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40175 (local address 127.0.98.2:46221)
0619 14:37:52.377768 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:52.377772 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:52.378051 (+ 279us) server_negotiation.cc:407] Connection header received
0619 14:37:52.378131 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:52.378134 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:52.378181 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:52.378240 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:37:52.378564 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:52.379162 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:52.380584 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:52.381043 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:52.381134 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:52.381200 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:52.381488 (+ 288us) server_negotiation.cc:299] Negotiation successful
0619 14:37:52.381558 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 14:37:52.381623 7708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9f7fbb6d1b04ae98dad245c92e55cec" candidate_uuid: "57ffea6d72ae469491ce019f5e03a885" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77032b2db22744feb997087f373b1209" is_pre_election: true
I20260619 14:37:52.381922 7708 raft_consensus.cc:2454] T b9f7fbb6d1b04ae98dad245c92e55cec P 77032b2db22744feb997087f373b1209 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57ffea6d72ae469491ce019f5e03a885 in term 0.
I20260619 14:37:52.382236 7572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9f7fbb6d1b04ae98dad245c92e55cec" candidate_uuid: "57ffea6d72ae469491ce019f5e03a885" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24dc9c1322cb4b5092de8d6a4c742d77" is_pre_election: true
I20260619 14:37:52.382388 7368 leader_election.cc:302] T b9f7fbb6d1b04ae98dad245c92e55cec P 57ffea6d72ae469491ce019f5e03a885 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57ffea6d72ae469491ce019f5e03a885, 77032b2db22744feb997087f373b1209; no voters:
I20260619 14:37:52.382489 7572 raft_consensus.cc:2454] T b9f7fbb6d1b04ae98dad245c92e55cec P 24dc9c1322cb4b5092de8d6a4c742d77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57ffea6d72ae469491ce019f5e03a885 in term 0.
I20260619 14:37:52.382663 7796 raft_consensus.cc:2781] T b9f7fbb6d1b04ae98dad245c92e55cec P 57ffea6d72ae469491ce019f5e03a885 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:37:52.382759 7796 raft_consensus.cc:482] T b9f7fbb6d1b04ae98dad245c92e55cec P 57ffea6d72ae469491ce019f5e03a885 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:37:52.382803 7796 raft_consensus.cc:3037] T b9f7fbb6d1b04ae98dad245c92e55cec P 57ffea6d72ae469491ce019f5e03a885 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:52.383793 7796 raft_consensus.cc:504] T b9f7fbb6d1b04ae98dad245c92e55cec P 57ffea6d72ae469491ce019f5e03a885 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ffea6d72ae469491ce019f5e03a885" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33487 } } peers { permanent_uuid: "77032b2db22744feb997087f373b1209" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36403 } } peers { permanent_uuid: "24dc9c1322cb4b5092de8d6a4c742d77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46221 } }
I20260619 14:37:52.384096 7796 leader_election.cc:288] T b9f7fbb6d1b04ae98dad245c92e55cec P 57ffea6d72ae469491ce019f5e03a885 [CANDIDATE]: Term 1 election: Requested vote from peers 77032b2db22744feb997087f373b1209 (127.0.98.3:36403), 24dc9c1322cb4b5092de8d6a4c742d77 (127.0.98.2:46221)
I20260619 14:37:52.384390 7572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9f7fbb6d1b04ae98dad245c92e55cec" candidate_uuid: "57ffea6d72ae469491ce019f5e03a885" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24dc9c1322cb4b5092de8d6a4c742d77"
I20260619 14:37:52.384392 7708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9f7fbb6d1b04ae98dad245c92e55cec" candidate_uuid: "57ffea6d72ae469491ce019f5e03a885" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77032b2db22744feb997087f373b1209"
I20260619 14:37:52.384487 7572 raft_consensus.cc:3037] T b9f7fbb6d1b04ae98dad245c92e55cec P 24dc9c1322cb4b5092de8d6a4c742d77 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:52.384493 7708 raft_consensus.cc:3037] T b9f7fbb6d1b04ae98dad245c92e55cec P 77032b2db22744feb997087f373b1209 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:52.385488 7572 raft_consensus.cc:2454] T b9f7fbb6d1b04ae98dad245c92e55cec P 24dc9c1322cb4b5092de8d6a4c742d77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57ffea6d72ae469491ce019f5e03a885 in term 1.
I20260619 14:37:52.385739 7369 leader_election.cc:302] T b9f7fbb6d1b04ae98dad245c92e55cec P 57ffea6d72ae469491ce019f5e03a885 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24dc9c1322cb4b5092de8d6a4c742d77, 57ffea6d72ae469491ce019f5e03a885; no voters:
I20260619 14:37:52.385921 7708 raft_consensus.cc:2454] T b9f7fbb6d1b04ae98dad245c92e55cec P 77032b2db22744feb997087f373b1209 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57ffea6d72ae469491ce019f5e03a885 in term 1.
I20260619 14:37:52.385926 7796 raft_consensus.cc:2781] T b9f7fbb6d1b04ae98dad245c92e55cec P 57ffea6d72ae469491ce019f5e03a885 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:37:52.386116 7796 raft_consensus.cc:686] T b9f7fbb6d1b04ae98dad245c92e55cec P 57ffea6d72ae469491ce019f5e03a885 [term 1 LEADER]: Becoming Leader. State: Replica: 57ffea6d72ae469491ce019f5e03a885, State: Running, Role: LEADER
I20260619 14:37:52.386436 7796 consensus_queue.cc:237] T b9f7fbb6d1b04ae98dad245c92e55cec P 57ffea6d72ae469491ce019f5e03a885 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ffea6d72ae469491ce019f5e03a885" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33487 } } peers { permanent_uuid: "77032b2db22744feb997087f373b1209" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36403 } } peers { permanent_uuid: "24dc9c1322cb4b5092de8d6a4c742d77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46221 } }
I20260619 14:37:52.389221 6512 catalog_manager.cc:5495] T b9f7fbb6d1b04ae98dad245c92e55cec P 57ffea6d72ae469491ce019f5e03a885 reported cstate change: term changed from 0 to 1, leader changed from <none> to 57ffea6d72ae469491ce019f5e03a885 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "57ffea6d72ae469491ce019f5e03a885" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57ffea6d72ae469491ce019f5e03a885" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77032b2db22744feb997087f373b1209" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24dc9c1322cb4b5092de8d6a4c742d77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46221 } health_report { overall_health: UNKNOWN } } }
I20260619 14:37:52.409442 7788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:52.405949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43680 (local address 127.0.98.1:33487)
0619 14:37:52.406067 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:52.406071 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:52.406105 (+ 34us) server_negotiation.cc:407] Connection header received
0619 14:37:52.406256 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:52.406258 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:52.406316 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:52.406397 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:37:52.406702 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:52.407302 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:52.408152 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:52.408370 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:52.408444 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:52.408668 (+ 224us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:37:52.409101 (+ 433us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:37:52.409190 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:52.409258 (+ 68us) server_negotiation.cc:299] Negotiation successful
0619 14:37:52.409303 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 14:37:52.415091 7708 raft_consensus.cc:1261] T b9f7fbb6d1b04ae98dad245c92e55cec P 77032b2db22744feb997087f373b1209 [term 1 FOLLOWER]: Refusing update from remote peer 57ffea6d72ae469491ce019f5e03a885: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:37:52.415089 7572 raft_consensus.cc:1261] T b9f7fbb6d1b04ae98dad245c92e55cec P 24dc9c1322cb4b5092de8d6a4c742d77 [term 1 FOLLOWER]: Refusing update from remote peer 57ffea6d72ae469491ce019f5e03a885: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:37:52.415751 7796 consensus_queue.cc:1035] T b9f7fbb6d1b04ae98dad245c92e55cec P 57ffea6d72ae469491ce019f5e03a885 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24dc9c1322cb4b5092de8d6a4c742d77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:37:52.415921 7798 consensus_queue.cc:1035] T b9f7fbb6d1b04ae98dad245c92e55cec P 57ffea6d72ae469491ce019f5e03a885 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77032b2db22744feb997087f373b1209" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:37:52.433368 6508 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:37:52.433535 6508 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:37:52.437309 6508 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e41b19ce379e45198aeff6fd3814bb43: Sending DeleteTablet for 3 replicas of tablet b9f7fbb6d1b04ae98dad245c92e55cec
I20260619 14:37:52.438663 7415 tablet_service.cc:1514] Processing DeleteTablet for tablet b9f7fbb6d1b04ae98dad245c92e55cec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:37:52 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43668
I20260619 14:37:52.438737 7688 tablet_service.cc:1514] Processing DeleteTablet for tablet b9f7fbb6d1b04ae98dad245c92e55cec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:37:52 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34938
I20260619 14:37:52.439060 7552 tablet_service.cc:1514] Processing DeleteTablet for tablet b9f7fbb6d1b04ae98dad245c92e55cec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:37:52 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49850
I20260619 14:37:52.439488 7812 tablet_replica.cc:330] T b9f7fbb6d1b04ae98dad245c92e55cec P 24dc9c1322cb4b5092de8d6a4c742d77: stopping tablet replica
I20260619 14:37:52.439883 7812 raft_consensus.cc:2229] T b9f7fbb6d1b04ae98dad245c92e55cec P 24dc9c1322cb4b5092de8d6a4c742d77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:37:52.439922 7810 tablet_replica.cc:330] T b9f7fbb6d1b04ae98dad245c92e55cec P 57ffea6d72ae469491ce019f5e03a885: stopping tablet replica
I20260619 14:37:52.440199 7812 raft_consensus.cc:2258] T b9f7fbb6d1b04ae98dad245c92e55cec P 24dc9c1322cb4b5092de8d6a4c742d77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:37:52.440215 7810 raft_consensus.cc:2229] T b9f7fbb6d1b04ae98dad245c92e55cec P 57ffea6d72ae469491ce019f5e03a885 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:37:52.440575 7810 raft_consensus.cc:2258] T b9f7fbb6d1b04ae98dad245c92e55cec P 57ffea6d72ae469491ce019f5e03a885 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:37:52.441576 7811 tablet_replica.cc:330] T b9f7fbb6d1b04ae98dad245c92e55cec P 77032b2db22744feb997087f373b1209: stopping tablet replica
I20260619 14:37:52.441684 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 6480
I20260619 14:37:52.442446 7811 raft_consensus.cc:2229] T b9f7fbb6d1b04ae98dad245c92e55cec P 77032b2db22744feb997087f373b1209 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:37:52.442749 7811 raft_consensus.cc:2258] T b9f7fbb6d1b04ae98dad245c92e55cec P 77032b2db22744feb997087f373b1209 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:37:52.443886 7812 ts_tablet_manager.cc:1905] T b9f7fbb6d1b04ae98dad245c92e55cec P 24dc9c1322cb4b5092de8d6a4c742d77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:37:52.445931 7810 ts_tablet_manager.cc:1905] T b9f7fbb6d1b04ae98dad245c92e55cec P 57ffea6d72ae469491ce019f5e03a885: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:37:52.446475 7812 ts_tablet_manager.cc:1918] T b9f7fbb6d1b04ae98dad245c92e55cec P 24dc9c1322cb4b5092de8d6a4c742d77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:37:52.446568 7812 log.cc:1196] T b9f7fbb6d1b04ae98dad245c92e55cec P 24dc9c1322cb4b5092de8d6a4c742d77: Deleting WAL directory at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-1/wal/wals/b9f7fbb6d1b04ae98dad245c92e55cec
I20260619 14:37:52.446862 7812 ts_tablet_manager.cc:1939] T b9f7fbb6d1b04ae98dad245c92e55cec P 24dc9c1322cb4b5092de8d6a4c742d77: Deleting consensus metadata
I20260619 14:37:52.448983 7810 ts_tablet_manager.cc:1918] T b9f7fbb6d1b04ae98dad245c92e55cec P 57ffea6d72ae469491ce019f5e03a885: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:37:52.449137 7810 log.cc:1196] T b9f7fbb6d1b04ae98dad245c92e55cec P 57ffea6d72ae469491ce019f5e03a885: Deleting WAL directory at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-0/wal/wals/b9f7fbb6d1b04ae98dad245c92e55cec
I20260619 14:37:52.449232 7811 ts_tablet_manager.cc:1905] T b9f7fbb6d1b04ae98dad245c92e55cec P 77032b2db22744feb997087f373b1209: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:37:52.449530 7810 ts_tablet_manager.cc:1939] T b9f7fbb6d1b04ae98dad245c92e55cec P 57ffea6d72ae469491ce019f5e03a885: Deleting consensus metadata
I20260619 14:37:52.451615 7811 ts_tablet_manager.cc:1918] T b9f7fbb6d1b04ae98dad245c92e55cec P 77032b2db22744feb997087f373b1209: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:37:52.451704 7811 log.cc:1196] T b9f7fbb6d1b04ae98dad245c92e55cec P 77032b2db22744feb997087f373b1209: Deleting WAL directory at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-2/wal/wals/b9f7fbb6d1b04ae98dad245c92e55cec
I20260619 14:37:52.451984 7811 ts_tablet_manager.cc:1939] T b9f7fbb6d1b04ae98dad245c92e55cec P 77032b2db22744feb997087f373b1209: Deleting consensus metadata
W20260619 14:37:52.453279 7641 connection.cc:535] server connection from 127.0.0.1:34938 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260619 14:37:52.453688 7505 connection.cc:535] server connection from 127.0.0.1:49850 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260619 14:37:52.454068 7371 connection.cc:535] server connection from 127.0.0.1:43668 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 14:37:52.462586 7770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:52.441095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47456 (local address 127.0.98.61:39319)
0619 14:37:52.441208 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:52.441212 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:52.453306 (+ 12094us) server_negotiation.cc:407] Connection header received
0619 14:37:52.453501 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:52.453504 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:52.453549 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:52.453593 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:52.454547 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:52.455349 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:52.456780 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:52.457156 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:52.457225 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:52.459221 (+ 1996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:52.459225 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:52.459228 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:52.459256 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:52.460837 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:52.461333 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:52.461336 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:52.461338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:52.461416 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:52.461677 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:52.461680 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:52.461682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:52.461886 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:52.461984 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:52.462319 (+ 335us) server_negotiation.cc:299] Negotiation successful
0619 14:37:52.462437 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 14:37:52.470564 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:52.458770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57606 (local address 127.0.98.60:39349)
0619 14:37:52.458887 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:52.458891 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:52.460002 (+ 1111us) server_negotiation.cc:407] Connection header received
0619 14:37:52.460825 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:52.460829 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:52.460884 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:52.460934 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:52.462767 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:52.463572 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:52.464912 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:52.465281 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:52.465385 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:52.467093 (+ 1708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:52.467098 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:52.467101 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:52.467127 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:52.468780 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:52.469223 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:52.469226 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:52.469228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:52.469322 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:52.469623 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:52.469626 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:52.469628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:52.469829 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:52.469932 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:52.470268 (+ 336us) server_negotiation.cc:299] Negotiation successful
0619 14:37:52.470402 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 14:37:52.471551 7486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:52.441460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57598 (local address 127.0.98.60:39349)
0619 14:37:52.441583 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:52.441588 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:52.463121 (+ 21533us) server_negotiation.cc:407] Connection header received
0619 14:37:52.463276 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:52.463280 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:52.463326 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:52.463374 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:52.464173 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:52.464767 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:52.465850 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:52.466158 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:52.466229 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:52.468218 (+ 1989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:52.468221 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:52.468224 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:52.468251 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:52.469986 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:52.470379 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:52.470383 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:52.470384 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:52.470452 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:52.470733 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:52.470736 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:52.470738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:52.470934 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:52.471027 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:52.471328 (+ 301us) server_negotiation.cc:299] Negotiation successful
0619 14:37:52.471423 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 14:37:52.472572 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:52.458094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47464 (local address 127.0.98.61:39319)
0619 14:37:52.458205 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:52.458209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:52.459658 (+ 1449us) server_negotiation.cc:407] Connection header received
0619 14:37:52.460146 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:52.460149 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:52.460209 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:52.460258 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:52.465026 (+ 4768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:52.465771 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:52.467015 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:52.467347 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:52.467412 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:52.468915 (+ 1503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:52.468919 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:52.468922 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:52.468945 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:52.470506 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:52.471209 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:52.471213 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:52.471215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:52.471292 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:52.471564 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:52.471568 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:52.471570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:52.471761 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:52.471880 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:52.472301 (+ 421us) server_negotiation.cc:299] Negotiation successful
0619 14:37:52.472437 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 14:37:53.430446 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:53.429696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46665 (local address 127.0.98.3:52465)
0619 14:37:53.430191 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:53.430285 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":70,"threads_started":1}
W20260619 14:37:53.430768 7755 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46665 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:46665: connect: Connection refused (error 111)
I20260619 14:37:53.430943 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:53.430513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46665 (local address 127.0.98.2:52771)
0619 14:37:53.430748 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:53.430852 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":57,"threads_started":1}
W20260619 14:37:53.431185 7619 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46665 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:46665: connect: Connection refused (error 111)
I20260619 14:37:53.431437 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:53.430159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46665 (local address 127.0.98.1:54671)
0619 14:37:53.431258 (+ 1099us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:53.431352 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":814,"thread_start_us":63,"threads_started":1}
W20260619 14:37:53.431746 7482 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46665 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:46665: connect: Connection refused (error 111)
I20260619 14:37:53.937935 7824 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e41b19ce379e45198aeff6fd3814bb43)
I20260619 14:37:53.938136 7824 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e41b19ce379e45198aeff6fd3814bb43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46665 } } peers { permanent_uuid: "6731bc26510d493c99ecc1aba97fd05b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39319 } attrs { promote: false } } peers { permanent_uuid: "827bf26a4cd440e59c8db36bbc95d991" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39349 } attrs { promote: false } }
I20260619 14:37:53.938264 7825 raft_consensus.cc:482] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e41b19ce379e45198aeff6fd3814bb43)
I20260619 14:37:53.938472 7825 raft_consensus.cc:504] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e41b19ce379e45198aeff6fd3814bb43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46665 } } peers { permanent_uuid: "6731bc26510d493c99ecc1aba97fd05b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39319 } attrs { promote: false } } peers { permanent_uuid: "827bf26a4cd440e59c8db36bbc95d991" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39349 } attrs { promote: false } }
I20260619 14:37:53.939149 7824 leader_election.cc:288] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e41b19ce379e45198aeff6fd3814bb43 (127.0.98.62:46665), 827bf26a4cd440e59c8db36bbc95d991 (127.0.98.60:39349)
I20260619 14:37:53.939854 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:53.939359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46665 (local address 127.0.0.1:48874)
0619 14:37:53.939670 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:53.939747 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":54,"threads_started":1}
I20260619 14:37:53.939885 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:53.939360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46665 (local address 127.0.0.1:48872)
0619 14:37:53.939720 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:53.939795 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":66,"threads_started":1}
I20260619 14:37:53.940191 7825 leader_election.cc:288] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e41b19ce379e45198aeff6fd3814bb43 (127.0.98.62:46665), 6731bc26510d493c99ecc1aba97fd05b (127.0.98.61:39319)
W20260619 14:37:53.940189 7106 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46665: connect: Connection refused (error 111)
W20260619 14:37:53.940582 7289 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46665: connect: Connection refused (error 111)
I20260619 14:37:53.941082 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:53.940845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46665 (local address 127.0.0.1:48886)
0619 14:37:53.940974 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:53.941017 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260619 14:37:53.941210 7106 leader_election.cc:334] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e41b19ce379e45198aeff6fd3814bb43 (127.0.98.62:46665): Network error: Client connection negotiation failed: client connection to 127.0.98.62:46665: connect: Connection refused (error 111)
I20260619 14:37:53.941474 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:53.941229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46665 (local address 127.0.0.1:48892)
0619 14:37:53.941363 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:53.941410 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260619 14:37:53.941594 7289 leader_election.cc:334] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e41b19ce379e45198aeff6fd3814bb43 (127.0.98.62:46665): Network error: Client connection negotiation failed: client connection to 127.0.98.62:46665: connect: Connection refused (error 111)
I20260619 14:37:53.943442 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:53.939754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39319 (local address 127.0.0.1:47472)
0619 14:37:53.939996 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:53.940009 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:53.940100 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:53.940418 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:53.940421 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:53.940442 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:37:53.940546 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:53.940554 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:53.941549 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:53.941552 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:53.942403 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:53.942412 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:53.943038 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:53.943041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:53.943158 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:53.943181 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 14:37:53.943246 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 14:37:53.943315 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":29,"thread_start_us":46,"threads_started":1}
I20260619 14:37:53.943445 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:53.939983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47472 (local address 127.0.98.61:39319)
0619 14:37:53.940193 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:53.940197 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:53.940209 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:37:53.940258 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:53.940261 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:53.940304 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:53.940390 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:37:53.940697 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:53.941447 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:53.942517 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:53.942939 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:53.943054 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:53.943115 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:53.943266 (+ 151us) server_negotiation.cc:299] Negotiation successful
0619 14:37:53.943346 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":64,"threads_started":1}
I20260619 14:37:53.943890 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:53.939526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57620 (local address 127.0.98.60:39349)
0619 14:37:53.940196 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:53.940200 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:53.940211 (+ 11us) server_negotiation.cc:407] Connection header received
0619 14:37:53.940248 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:53.940251 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:53.940371 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:53.940776 (+ 405us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:37:53.940864 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:53.941812 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:53.942832 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:53.943299 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:53.943388 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:53.943439 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:53.943718 (+ 279us) server_negotiation.cc:299] Negotiation successful
0619 14:37:53.943791 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":612,"thread_start_us":48,"threads_started":1}
I20260619 14:37:53.943888 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:53.939373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39349 (local address 127.0.0.1:57620)
0619 14:37:53.939787 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:53.939800 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:53.940046 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:53.940473 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:53.940476 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:53.940490 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:37:53.940568 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:53.940574 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:53.941908 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:53.941911 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:53.942726 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:53.942735 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:53.943399 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:53.943402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:53.943491 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:53.943634 (+ 143us) client_negotiation.cc:769] Sending connection context
0619 14:37:53.943701 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 14:37:53.943775 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":65,"threads_started":1}
I20260619 14:37:53.943935 7129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "827bf26a4cd440e59c8db36bbc95d991" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6731bc26510d493c99ecc1aba97fd05b" is_pre_election: true
I20260619 14:37:53.944128 7129 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 827bf26a4cd440e59c8db36bbc95d991 in term 1.
I20260619 14:37:53.944438 7312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6731bc26510d493c99ecc1aba97fd05b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "827bf26a4cd440e59c8db36bbc95d991" is_pre_election: true
I20260619 14:37:53.944447 7289 leader_election.cc:302] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6731bc26510d493c99ecc1aba97fd05b, 827bf26a4cd440e59c8db36bbc95d991; no voters: e41b19ce379e45198aeff6fd3814bb43
I20260619 14:37:53.944625 7312 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6731bc26510d493c99ecc1aba97fd05b in term 1.
I20260619 14:37:53.944701 7825 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 14:37:53.944782 7825 raft_consensus.cc:482] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e41b19ce379e45198aeff6fd3814bb43)
I20260619 14:37:53.944828 7825 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991 [term 1 FOLLOWER]: Advancing to term 2
I20260619 14:37:53.944958 7104 leader_election.cc:302] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6731bc26510d493c99ecc1aba97fd05b, 827bf26a4cd440e59c8db36bbc95d991; no voters: e41b19ce379e45198aeff6fd3814bb43
I20260619 14:37:53.945151 7824 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 14:37:53.945204 7824 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b [term 1 FOLLOWER]: Starting leader election (detected failure of leader e41b19ce379e45198aeff6fd3814bb43)
I20260619 14:37:53.945252 7824 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b [term 1 FOLLOWER]: Advancing to term 2
I20260619 14:37:53.945940 7825 raft_consensus.cc:504] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e41b19ce379e45198aeff6fd3814bb43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46665 } } peers { permanent_uuid: "6731bc26510d493c99ecc1aba97fd05b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39319 } attrs { promote: false } } peers { permanent_uuid: "827bf26a4cd440e59c8db36bbc95d991" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39349 } attrs { promote: false } }
I20260619 14:37:53.946260 7825 leader_election.cc:288] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991 [CANDIDATE]: Term 2 election: Requested vote from peers e41b19ce379e45198aeff6fd3814bb43 (127.0.98.62:46665), 6731bc26510d493c99ecc1aba97fd05b (127.0.98.61:39319)
I20260619 14:37:53.946465 7824 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e41b19ce379e45198aeff6fd3814bb43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46665 } } peers { permanent_uuid: "6731bc26510d493c99ecc1aba97fd05b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39319 } attrs { promote: false } } peers { permanent_uuid: "827bf26a4cd440e59c8db36bbc95d991" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39349 } attrs { promote: false } }
I20260619 14:37:53.946620 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:53.946417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46665 (local address 127.0.0.1:48906)
0619 14:37:53.946511 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:53.946559 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260619 14:37:53.946863 7824 leader_election.cc:288] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b [CANDIDATE]: Term 2 election: Requested vote from peers e41b19ce379e45198aeff6fd3814bb43 (127.0.98.62:46665), 827bf26a4cd440e59c8db36bbc95d991 (127.0.98.60:39349)
I20260619 14:37:53.947011 7129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "827bf26a4cd440e59c8db36bbc95d991" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6731bc26510d493c99ecc1aba97fd05b"
I20260619 14:37:53.947175 7129 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 827bf26a4cd440e59c8db36bbc95d991 in current term 2: Already voted for candidate 6731bc26510d493c99ecc1aba97fd05b in this term.
I20260619 14:37:53.947196 7312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6731bc26510d493c99ecc1aba97fd05b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "827bf26a4cd440e59c8db36bbc95d991"
I20260619 14:37:53.947357 7312 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 6731bc26510d493c99ecc1aba97fd05b in current term 2: Already voted for candidate 827bf26a4cd440e59c8db36bbc95d991 in this term.
I20260619 14:37:53.947484 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:53.946906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46665 (local address 127.0.0.1:48910)
0619 14:37:53.947082 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:53.947422 (+ 340us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260619 14:37:53.947561 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:53.947109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46665 (local address 127.0.0.1:48916)
0619 14:37:53.947275 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:53.947347 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
W20260619 14:37:53.947636 7289 leader_election.cc:334] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e41b19ce379e45198aeff6fd3814bb43 (127.0.98.62:46665): Network error: Client connection negotiation failed: client connection to 127.0.98.62:46665: connect: Connection refused (error 111)
I20260619 14:37:53.947692 7289 leader_election.cc:302] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 827bf26a4cd440e59c8db36bbc95d991; no voters: 6731bc26510d493c99ecc1aba97fd05b, e41b19ce379e45198aeff6fd3814bb43
I20260619 14:37:53.947994 7825 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260619 14:37:53.948251 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:53.948063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46665 (local address 127.0.0.1:48918)
0619 14:37:53.948162 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:53.948202 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260619 14:37:53.948346 7106 leader_election.cc:334] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e41b19ce379e45198aeff6fd3814bb43 (127.0.98.62:46665): Network error: Client connection negotiation failed: client connection to 127.0.98.62:46665: connect: Connection refused (error 111)
I20260619 14:37:53.948415 7106 leader_election.cc:302] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6731bc26510d493c99ecc1aba97fd05b; no voters: 827bf26a4cd440e59c8db36bbc95d991, e41b19ce379e45198aeff6fd3814bb43
I20260619 14:37:53.948532 7824 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260619 14:37:54.432142 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:54.431302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46665 (local address 127.0.98.3:44939)
0619 14:37:54.431706 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:54.431826 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":99,"threads_started":1}
I20260619 14:37:54.432562 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:54.432197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46665 (local address 127.0.98.1:45451)
0619 14:37:54.432416 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:54.432476 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":87,"threads_started":1}
I20260619 14:37:54.432686 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:54.431672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46665 (local address 127.0.98.2:33867)
0619 14:37:54.432525 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:54.432584 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":782,"thread_start_us":109,"threads_started":1}
I20260619 14:37:55.433542 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:55.433056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46665 (local address 127.0.98.3:40531)
0619 14:37:55.433369 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:55.433434 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":75,"threads_started":1}
I20260619 14:37:55.433724 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:55.433361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46665 (local address 127.0.98.2:49393)
0619 14:37:55.433590 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:55.433642 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":55,"threads_started":1}
I20260619 14:37:55.433923 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:55.433138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46665 (local address 127.0.98.1:33639)
0619 14:37:55.433787 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:55.433843 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":79,"threads_started":1}
I20260619 14:37:55.526988 7840 raft_consensus.cc:482] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:37:55.527117 7840 raft_consensus.cc:504] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e41b19ce379e45198aeff6fd3814bb43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46665 } } peers { permanent_uuid: "6731bc26510d493c99ecc1aba97fd05b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39319 } attrs { promote: false } } peers { permanent_uuid: "827bf26a4cd440e59c8db36bbc95d991" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39349 } attrs { promote: false } }
I20260619 14:37:55.527513 7840 leader_election.cc:288] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e41b19ce379e45198aeff6fd3814bb43 (127.0.98.62:46665), 6731bc26510d493c99ecc1aba97fd05b (127.0.98.61:39319)
I20260619 14:37:55.528113 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:55.527720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46665 (local address 127.0.0.1:49158)
0619 14:37:55.527959 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:55.528027 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":61,"threads_started":1}
I20260619 14:37:55.528239 7129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "827bf26a4cd440e59c8db36bbc95d991" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6731bc26510d493c99ecc1aba97fd05b" is_pre_election: true
I20260619 14:37:55.528421 7129 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 827bf26a4cd440e59c8db36bbc95d991 in term 2.
I20260619 14:37:55.528884 7289 leader_election.cc:302] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6731bc26510d493c99ecc1aba97fd05b, 827bf26a4cd440e59c8db36bbc95d991; no voters:
I20260619 14:37:55.528980 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:55.528706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46665 (local address 127.0.0.1:49170)
0619 14:37:55.528854 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:55.528910 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260619 14:37:55.529047 7840 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260619 14:37:55.529103 7840 raft_consensus.cc:482] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260619 14:37:55.529111 7289 leader_election.cc:334] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e41b19ce379e45198aeff6fd3814bb43 (127.0.98.62:46665): Network error: Client connection negotiation failed: client connection to 127.0.98.62:46665: connect: Connection refused (error 111)
I20260619 14:37:55.529148 7840 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991 [term 2 FOLLOWER]: Advancing to term 3
I20260619 14:37:55.530129 7840 raft_consensus.cc:504] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e41b19ce379e45198aeff6fd3814bb43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46665 } } peers { permanent_uuid: "6731bc26510d493c99ecc1aba97fd05b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39319 } attrs { promote: false } } peers { permanent_uuid: "827bf26a4cd440e59c8db36bbc95d991" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39349 } attrs { promote: false } }
I20260619 14:37:55.530416 7840 leader_election.cc:288] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991 [CANDIDATE]: Term 3 election: Requested vote from peers e41b19ce379e45198aeff6fd3814bb43 (127.0.98.62:46665), 6731bc26510d493c99ecc1aba97fd05b (127.0.98.61:39319)
I20260619 14:37:55.530759 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:55.530546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46665 (local address 127.0.0.1:49182)
0619 14:37:55.530640 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:55.530708 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260619 14:37:55.530902 7129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "827bf26a4cd440e59c8db36bbc95d991" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6731bc26510d493c99ecc1aba97fd05b"
I20260619 14:37:55.531029 7129 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b [term 2 FOLLOWER]: Advancing to term 3
I20260619 14:37:55.531340 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:55.531170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46665 (local address 127.0.0.1:49198)
0619 14:37:55.531254 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:55.531293 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260619 14:37:55.531432 7289 leader_election.cc:334] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer e41b19ce379e45198aeff6fd3814bb43 (127.0.98.62:46665): Network error: Client connection negotiation failed: client connection to 127.0.98.62:46665: connect: Connection refused (error 111)
I20260619 14:37:55.532140 7129 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 827bf26a4cd440e59c8db36bbc95d991 in term 3.
I20260619 14:37:55.532405 7289 leader_election.cc:302] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6731bc26510d493c99ecc1aba97fd05b, 827bf26a4cd440e59c8db36bbc95d991; no voters: e41b19ce379e45198aeff6fd3814bb43
I20260619 14:37:55.532568 7840 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 14:37:55.532774 7840 raft_consensus.cc:686] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991 [term 3 LEADER]: Becoming Leader. State: Replica: 827bf26a4cd440e59c8db36bbc95d991, State: Running, Role: LEADER
I20260619 14:37:55.533031 7840 consensus_queue.cc:237] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e41b19ce379e45198aeff6fd3814bb43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46665 } } peers { permanent_uuid: "6731bc26510d493c99ecc1aba97fd05b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39319 } attrs { promote: false } } peers { permanent_uuid: "827bf26a4cd440e59c8db36bbc95d991" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39349 } attrs { promote: false } }
I20260619 14:37:55.534271 7843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 827bf26a4cd440e59c8db36bbc95d991. Latest consensus state: current_term: 3 leader_uuid: "827bf26a4cd440e59c8db36bbc95d991" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e41b19ce379e45198aeff6fd3814bb43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46665 } } peers { permanent_uuid: "6731bc26510d493c99ecc1aba97fd05b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39319 } attrs { promote: false } } peers { permanent_uuid: "827bf26a4cd440e59c8db36bbc95d991" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39349 } attrs { promote: false } } }
I20260619 14:37:55.534394 7843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991 [sys.catalog]: This master's current role is: LEADER
I20260619 14:37:55.534662 7845 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:37:55.536664 7845 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:37:55.537081 7845 catalog_manager.cc:1259] Loaded cluster ID: a546ae03a7f4400d99fa082375d31b43
I20260619 14:37:55.537127 7845 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:37:55.537647 7845 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:37:55.538014 7845 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991: Loaded TSK: 0
I20260619 14:37:55.538688 7845 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:37:55.597873 7301 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57606:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:37:55.600847 7301 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:37:55.603696 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:55.603477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46665 (local address 127.0.0.1:49210)
0619 14:37:55.603588 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:55.603632 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 14:37:55.603677 7129 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b [term 3 FOLLOWER]: Refusing update from remote peer 827bf26a4cd440e59c8db36bbc95d991: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260619 14:37:55.604085 7843 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6731bc26510d493c99ecc1aba97fd05b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260619 14:37:55.604379 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:55.604162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46665 (local address 127.0.0.1:49218)
0619 14:37:55.604266 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:55.604317 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260619 14:37:55.604565 7289 consensus_peers.cc:487] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991 -> Peer e41b19ce379e45198aeff6fd3814bb43 (127.0.98.62:46665): Couldn't send request to peer e41b19ce379e45198aeff6fd3814bb43. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46665: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 14:37:55.605440 7847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 827bf26a4cd440e59c8db36bbc95d991. Latest consensus state: current_term: 3 leader_uuid: "827bf26a4cd440e59c8db36bbc95d991" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e41b19ce379e45198aeff6fd3814bb43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46665 } } peers { permanent_uuid: "6731bc26510d493c99ecc1aba97fd05b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39319 } attrs { promote: false } } peers { permanent_uuid: "827bf26a4cd440e59c8db36bbc95d991" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39349 } attrs { promote: false } } }
I20260619 14:37:55.605547 7847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:37:55.606480 7843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "827bf26a4cd440e59c8db36bbc95d991" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e41b19ce379e45198aeff6fd3814bb43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46665 } } peers { permanent_uuid: "6731bc26510d493c99ecc1aba97fd05b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39319 } attrs { promote: false } } peers { permanent_uuid: "827bf26a4cd440e59c8db36bbc95d991" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39349 } attrs { promote: false } } }
I20260619 14:37:55.606586 7843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991 [sys.catalog]: This master's current role is: LEADER
I20260619 14:37:55.607669 7847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "827bf26a4cd440e59c8db36bbc95d991" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e41b19ce379e45198aeff6fd3814bb43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46665 } } peers { permanent_uuid: "6731bc26510d493c99ecc1aba97fd05b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39319 } attrs { promote: false } } peers { permanent_uuid: "827bf26a4cd440e59c8db36bbc95d991" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39349 } attrs { promote: false } } }
I20260619 14:37:55.607792 7847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6731bc26510d493c99ecc1aba97fd05b [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:37:55.615783 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:55.611763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36403 (local address 127.0.0.1:34954)
0619 14:37:55.611872 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:55.611887 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:55.611969 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:55.612591 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:55.612594 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:55.612612 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:37:55.612758 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:55.612765 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:55.613814 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:55.613817 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:55.614556 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:55.614566 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:55.615388 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:55.615391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:55.615505 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:55.615530 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 14:37:55.615577 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 14:37:55.615640 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":3,"spinlock_wait_cycles":7936}
I20260619 14:37:55.616405 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:55.611961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46221 (local address 127.0.0.1:49862)
0619 14:37:55.612547 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:55.612559 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:55.612641 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:55.612953 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:55.612956 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:55.612972 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:37:55.613082 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:55.613089 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:55.613995 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:55.613999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:55.615164 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:55.615173 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:55.616001 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:55.616005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:55.616101 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:55.616125 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 14:37:55.616199 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 14:37:55.616261 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"mutex_wait_us":1,"thread_start_us":72,"threads_started":1}
I20260619 14:37:55.616429 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:55.612250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49862 (local address 127.0.98.2:46221)
0619 14:37:55.612716 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:55.612744 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:55.612769 (+ 25us) server_negotiation.cc:407] Connection header received
0619 14:37:55.612817 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:55.612820 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:55.612866 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:55.612933 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:37:55.613181 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:55.613877 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:55.615327 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:55.615888 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:55.615993 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:55.616069 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:55.616218 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 14:37:55.616300 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":101,"threads_started":1}
I20260619 14:37:55.616707 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:55.612130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34954 (local address 127.0.98.3:36403)
0619 14:37:55.612351 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:55.612355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:55.612368 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:37:55.612418 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:55.612421 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:55.612466 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:55.612866 (+ 400us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:37:55.612976 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:55.613701 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:55.614654 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:55.615266 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:55.616219 (+ 953us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:55.616278 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:55.616311 (+ 33us) server_negotiation.cc:299] Negotiation successful
0619 14:37:55.616376 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":72,"threads_started":1}
I20260619 14:37:55.616899 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:55.612367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33487 (local address 127.0.0.1:43684)
0619 14:37:55.613257 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:55.613270 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:55.613358 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:55.613577 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:55.613580 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:55.613593 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:37:55.613666 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:55.613671 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:55.614577 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:55.614580 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:55.615624 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:55.615632 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:55.616497 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:55.616501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:55.616586 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:55.616626 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 14:37:55.616692 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 14:37:55.616782 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":837,"mutex_wait_us":198,"thread_start_us":74,"threads_started":1}
I20260619 14:37:55.617087 7552 tablet_service.cc:1467] Processing CreateTablet for tablet e84fab0f0799437e8ba2f84532c2cd03 (DEFAULT_TABLE table=test-table [id=186544586a75462e9475436082b11b6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:37:55.617349 7552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e84fab0f0799437e8ba2f84532c2cd03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:55.617308 7688 tablet_service.cc:1467] Processing CreateTablet for tablet e84fab0f0799437e8ba2f84532c2cd03 (DEFAULT_TABLE table=test-table [id=186544586a75462e9475436082b11b6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:37:55.617532 7688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e84fab0f0799437e8ba2f84532c2cd03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:55.619563 7859 tablet_bootstrap.cc:492] T e84fab0f0799437e8ba2f84532c2cd03 P 77032b2db22744feb997087f373b1209: Bootstrap starting.
I20260619 14:37:55.619581 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:55.612464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43684 (local address 127.0.98.1:33487)
0619 14:37:55.613137 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:55.613142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:55.613351 (+ 209us) server_negotiation.cc:407] Connection header received
0619 14:37:55.613435 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:55.613437 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:55.613486 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:55.613566 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:37:55.613761 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:55.614468 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:55.615745 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:55.616293 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:55.616380 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:55.616440 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:55.616709 (+ 269us) server_negotiation.cc:299] Negotiation successful
0619 14:37:55.616812 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":71,"threads_started":1}
I20260619 14:37:55.619885 7858 tablet_bootstrap.cc:492] T e84fab0f0799437e8ba2f84532c2cd03 P 24dc9c1322cb4b5092de8d6a4c742d77: Bootstrap starting.
I20260619 14:37:55.620083 7415 tablet_service.cc:1467] Processing CreateTablet for tablet e84fab0f0799437e8ba2f84532c2cd03 (DEFAULT_TABLE table=test-table [id=186544586a75462e9475436082b11b6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:37:55.620318 7415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e84fab0f0799437e8ba2f84532c2cd03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:55.620548 7859 tablet_bootstrap.cc:654] T e84fab0f0799437e8ba2f84532c2cd03 P 77032b2db22744feb997087f373b1209: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:55.620831 7858 tablet_bootstrap.cc:654] T e84fab0f0799437e8ba2f84532c2cd03 P 24dc9c1322cb4b5092de8d6a4c742d77: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:55.621634 7859 tablet_bootstrap.cc:492] T e84fab0f0799437e8ba2f84532c2cd03 P 77032b2db22744feb997087f373b1209: No bootstrap required, opened a new log
I20260619 14:37:55.621711 7859 ts_tablet_manager.cc:1397] T e84fab0f0799437e8ba2f84532c2cd03 P 77032b2db22744feb997087f373b1209: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 14:37:55.621896 7858 tablet_bootstrap.cc:492] T e84fab0f0799437e8ba2f84532c2cd03 P 24dc9c1322cb4b5092de8d6a4c742d77: No bootstrap required, opened a new log
I20260619 14:37:55.621979 7858 ts_tablet_manager.cc:1397] T e84fab0f0799437e8ba2f84532c2cd03 P 24dc9c1322cb4b5092de8d6a4c742d77: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 14:37:55.622107 7859 raft_consensus.cc:348] T e84fab0f0799437e8ba2f84532c2cd03 P 77032b2db22744feb997087f373b1209 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77032b2db22744feb997087f373b1209" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36403 } } peers { permanent_uuid: "24dc9c1322cb4b5092de8d6a4c742d77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46221 } } peers { permanent_uuid: "57ffea6d72ae469491ce019f5e03a885" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33487 } }
I20260619 14:37:55.622200 7859 raft_consensus.cc:374] T e84fab0f0799437e8ba2f84532c2cd03 P 77032b2db22744feb997087f373b1209 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:55.622388 7858 raft_consensus.cc:348] T e84fab0f0799437e8ba2f84532c2cd03 P 24dc9c1322cb4b5092de8d6a4c742d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77032b2db22744feb997087f373b1209" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36403 } } peers { permanent_uuid: "24dc9c1322cb4b5092de8d6a4c742d77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46221 } } peers { permanent_uuid: "57ffea6d72ae469491ce019f5e03a885" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33487 } }
I20260619 14:37:55.622447 7859 raft_consensus.cc:729] T e84fab0f0799437e8ba2f84532c2cd03 P 77032b2db22744feb997087f373b1209 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77032b2db22744feb997087f373b1209, State: Initialized, Role: FOLLOWER
I20260619 14:37:55.622535 7858 raft_consensus.cc:374] T e84fab0f0799437e8ba2f84532c2cd03 P 24dc9c1322cb4b5092de8d6a4c742d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:55.622594 7858 raft_consensus.cc:729] T e84fab0f0799437e8ba2f84532c2cd03 P 24dc9c1322cb4b5092de8d6a4c742d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24dc9c1322cb4b5092de8d6a4c742d77, State: Initialized, Role: FOLLOWER
I20260619 14:37:55.622725 7858 consensus_queue.cc:260] T e84fab0f0799437e8ba2f84532c2cd03 P 24dc9c1322cb4b5092de8d6a4c742d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77032b2db22744feb997087f373b1209" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36403 } } peers { permanent_uuid: "24dc9c1322cb4b5092de8d6a4c742d77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46221 } } peers { permanent_uuid: "57ffea6d72ae469491ce019f5e03a885" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33487 } }
I20260619 14:37:55.622948 7862 tablet_bootstrap.cc:492] T e84fab0f0799437e8ba2f84532c2cd03 P 57ffea6d72ae469491ce019f5e03a885: Bootstrap starting.
I20260619 14:37:55.622941 7859 consensus_queue.cc:260] T e84fab0f0799437e8ba2f84532c2cd03 P 77032b2db22744feb997087f373b1209 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77032b2db22744feb997087f373b1209" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36403 } } peers { permanent_uuid: "24dc9c1322cb4b5092de8d6a4c742d77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46221 } } peers { permanent_uuid: "57ffea6d72ae469491ce019f5e03a885" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33487 } }
I20260619 14:37:55.623107 7858 ts_tablet_manager.cc:1428] T e84fab0f0799437e8ba2f84532c2cd03 P 24dc9c1322cb4b5092de8d6a4c742d77: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:37:55.623569 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:55.623351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46665 (local address 127.0.98.2:43661)
0619 14:37:55.623449 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:55.623507 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260619 14:37:55.623916 7859 ts_tablet_manager.cc:1428] T e84fab0f0799437e8ba2f84532c2cd03 P 77032b2db22744feb997087f373b1209: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260619 14:37:55.624255 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:55.624013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46665 (local address 127.0.98.2:39965)
0619 14:37:55.624145 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:55.624190 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260619 14:37:55.624819 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:55.624563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46665 (local address 127.0.98.3:60399)
0619 14:37:55.624677 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:55.624757 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260619 14:37:55.625310 7618 heartbeater.cc:503] Master 127.0.98.60:39349 was elected leader, sending a full tablet report...
I20260619 14:37:55.625806 7754 heartbeater.cc:503] Master 127.0.98.60:39349 was elected leader, sending a full tablet report...
I20260619 14:37:55.626542 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:55.626286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46665 (local address 127.0.98.3:56327)
0619 14:37:55.626430 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:55.626476 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260619 14:37:55.626736 7862 tablet_bootstrap.cc:654] T e84fab0f0799437e8ba2f84532c2cd03 P 57ffea6d72ae469491ce019f5e03a885: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:55.627835 7862 tablet_bootstrap.cc:492] T e84fab0f0799437e8ba2f84532c2cd03 P 57ffea6d72ae469491ce019f5e03a885: No bootstrap required, opened a new log
I20260619 14:37:55.627907 7862 ts_tablet_manager.cc:1397] T e84fab0f0799437e8ba2f84532c2cd03 P 57ffea6d72ae469491ce019f5e03a885: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260619 14:37:55.628230 7862 raft_consensus.cc:348] T e84fab0f0799437e8ba2f84532c2cd03 P 57ffea6d72ae469491ce019f5e03a885 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77032b2db22744feb997087f373b1209" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36403 } } peers { permanent_uuid: "24dc9c1322cb4b5092de8d6a4c742d77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46221 } } peers { permanent_uuid: "57ffea6d72ae469491ce019f5e03a885" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33487 } }
I20260619 14:37:55.628340 7862 raft_consensus.cc:374] T e84fab0f0799437e8ba2f84532c2cd03 P 57ffea6d72ae469491ce019f5e03a885 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:55.628366 7862 raft_consensus.cc:729] T e84fab0f0799437e8ba2f84532c2cd03 P 57ffea6d72ae469491ce019f5e03a885 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57ffea6d72ae469491ce019f5e03a885, State: Initialized, Role: FOLLOWER
I20260619 14:37:55.628479 7862 consensus_queue.cc:260] T e84fab0f0799437e8ba2f84532c2cd03 P 57ffea6d72ae469491ce019f5e03a885 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77032b2db22744feb997087f373b1209" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36403 } } peers { permanent_uuid: "24dc9c1322cb4b5092de8d6a4c742d77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46221 } } peers { permanent_uuid: "57ffea6d72ae469491ce019f5e03a885" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33487 } }
I20260619 14:37:55.628789 7862 ts_tablet_manager.cc:1428] T e84fab0f0799437e8ba2f84532c2cd03 P 57ffea6d72ae469491ce019f5e03a885: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:37:55.629395 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:55.629190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46665 (local address 127.0.98.1:45281)
0619 14:37:55.629284 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:55.629333 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260619 14:37:55.629963 7481 heartbeater.cc:503] Master 127.0.98.60:39349 was elected leader, sending a full tablet report...
I20260619 14:37:55.630038 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:55.629830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46665 (local address 127.0.98.1:33983)
0619 14:37:55.629926 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:55.629973 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260619 14:37:55.648483 7863 raft_consensus.cc:482] T e84fab0f0799437e8ba2f84532c2cd03 P 24dc9c1322cb4b5092de8d6a4c742d77 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:37:55.648589 7863 raft_consensus.cc:504] T e84fab0f0799437e8ba2f84532c2cd03 P 24dc9c1322cb4b5092de8d6a4c742d77 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77032b2db22744feb997087f373b1209" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36403 } } peers { permanent_uuid: "24dc9c1322cb4b5092de8d6a4c742d77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46221 } } peers { permanent_uuid: "57ffea6d72ae469491ce019f5e03a885" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33487 } }
I20260619 14:37:55.649729 7863 leader_election.cc:288] T e84fab0f0799437e8ba2f84532c2cd03 P 24dc9c1322cb4b5092de8d6a4c742d77 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77032b2db22744feb997087f373b1209 (127.0.98.3:36403), 57ffea6d72ae469491ce019f5e03a885 (127.0.98.1:33487)
I20260619 14:37:55.653311 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:55.650001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43033 (local address 127.0.98.3:36403)
0619 14:37:55.650106 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:55.650110 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:55.650123 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:37:55.650201 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:55.650204 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:55.650252 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:55.650324 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:37:55.650633 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:55.651245 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:55.652323 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:55.652760 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:55.652847 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:55.652907 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:55.653125 (+ 218us) server_negotiation.cc:299] Negotiation successful
0619 14:37:55.653196 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 14:37:55.653321 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:55.650200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35935 (local address 127.0.98.1:33487)
0619 14:37:55.650318 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:55.650322 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:55.650507 (+ 185us) server_negotiation.cc:407] Connection header received
0619 14:37:55.650590 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:55.650593 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:55.650638 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:55.650692 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:37:55.650912 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:55.651424 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:55.652386 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:55.652809 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:55.652910 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:55.652967 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:55.653149 (+ 182us) server_negotiation.cc:299] Negotiation successful
0619 14:37:55.653217 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 14:37:55.653318 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:55.649907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36403 (local address 127.0.98.2:43033)
0619 14:37:55.650009 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:55.650022 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:55.650108 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:55.650384 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:55.650388 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:55.650408 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:37:55.650513 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:55.650520 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:55.651340 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:55.651344 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:55.652199 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:55.652207 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:55.652875 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:55.652879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:55.652974 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:55.653035 (+ 61us) client_negotiation.cc:769] Sending connection context
0619 14:37:55.653093 (+ 58us) client_negotiation.cc:240] Negotiation successful
0619 14:37:55.653161 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":19}
I20260619 14:37:55.653345 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:55.650133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33487 (local address 127.0.98.2:35935)
0619 14:37:55.650423 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:55.650435 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:55.650522 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:55.650709 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:55.650711 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:55.650727 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:37:55.650822 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:55.650828 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:55.651514 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:55.651517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:55.652263 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:55.652283 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:55.652919 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:55.652923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:55.653040 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:55.653057 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 14:37:55.653125 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 14:37:55.653190 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":18,"thread_start_us":122,"threads_started":1}
I20260619 14:37:55.654040 7708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e84fab0f0799437e8ba2f84532c2cd03" candidate_uuid: "24dc9c1322cb4b5092de8d6a4c742d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77032b2db22744feb997087f373b1209" is_pre_election: true
I20260619 14:37:55.654059 7435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e84fab0f0799437e8ba2f84532c2cd03" candidate_uuid: "24dc9c1322cb4b5092de8d6a4c742d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57ffea6d72ae469491ce019f5e03a885" is_pre_election: true
I20260619 14:37:55.654167 7708 raft_consensus.cc:2454] T e84fab0f0799437e8ba2f84532c2cd03 P 77032b2db22744feb997087f373b1209 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24dc9c1322cb4b5092de8d6a4c742d77 in term 0.
I20260619 14:37:55.654263 7435 raft_consensus.cc:2454] T e84fab0f0799437e8ba2f84532c2cd03 P 57ffea6d72ae469491ce019f5e03a885 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24dc9c1322cb4b5092de8d6a4c742d77 in term 0.
I20260619 14:37:55.654544 7505 leader_election.cc:302] T e84fab0f0799437e8ba2f84532c2cd03 P 24dc9c1322cb4b5092de8d6a4c742d77 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24dc9c1322cb4b5092de8d6a4c742d77, 77032b2db22744feb997087f373b1209; no voters:
I20260619 14:37:55.654809 7863 raft_consensus.cc:2781] T e84fab0f0799437e8ba2f84532c2cd03 P 24dc9c1322cb4b5092de8d6a4c742d77 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:37:55.654861 7863 raft_consensus.cc:482] T e84fab0f0799437e8ba2f84532c2cd03 P 24dc9c1322cb4b5092de8d6a4c742d77 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:37:55.654886 7863 raft_consensus.cc:3037] T e84fab0f0799437e8ba2f84532c2cd03 P 24dc9c1322cb4b5092de8d6a4c742d77 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:55.655902 7863 raft_consensus.cc:504] T e84fab0f0799437e8ba2f84532c2cd03 P 24dc9c1322cb4b5092de8d6a4c742d77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77032b2db22744feb997087f373b1209" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36403 } } peers { permanent_uuid: "24dc9c1322cb4b5092de8d6a4c742d77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46221 } } peers { permanent_uuid: "57ffea6d72ae469491ce019f5e03a885" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33487 } }
I20260619 14:37:55.656198 7863 leader_election.cc:288] T e84fab0f0799437e8ba2f84532c2cd03 P 24dc9c1322cb4b5092de8d6a4c742d77 [CANDIDATE]: Term 1 election: Requested vote from peers 77032b2db22744feb997087f373b1209 (127.0.98.3:36403), 57ffea6d72ae469491ce019f5e03a885 (127.0.98.1:33487)
I20260619 14:37:55.656490 7708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e84fab0f0799437e8ba2f84532c2cd03" candidate_uuid: "24dc9c1322cb4b5092de8d6a4c742d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77032b2db22744feb997087f373b1209"
I20260619 14:37:55.656540 7435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e84fab0f0799437e8ba2f84532c2cd03" candidate_uuid: "24dc9c1322cb4b5092de8d6a4c742d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57ffea6d72ae469491ce019f5e03a885"
I20260619 14:37:55.656598 7708 raft_consensus.cc:3037] T e84fab0f0799437e8ba2f84532c2cd03 P 77032b2db22744feb997087f373b1209 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:55.656651 7435 raft_consensus.cc:3037] T e84fab0f0799437e8ba2f84532c2cd03 P 57ffea6d72ae469491ce019f5e03a885 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:55.657630 7708 raft_consensus.cc:2454] T e84fab0f0799437e8ba2f84532c2cd03 P 77032b2db22744feb997087f373b1209 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24dc9c1322cb4b5092de8d6a4c742d77 in term 1.
I20260619 14:37:55.657706 7435 raft_consensus.cc:2454] T e84fab0f0799437e8ba2f84532c2cd03 P 57ffea6d72ae469491ce019f5e03a885 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24dc9c1322cb4b5092de8d6a4c742d77 in term 1.
I20260619 14:37:55.657927 7505 leader_election.cc:302] T e84fab0f0799437e8ba2f84532c2cd03 P 24dc9c1322cb4b5092de8d6a4c742d77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24dc9c1322cb4b5092de8d6a4c742d77, 77032b2db22744feb997087f373b1209; no voters:
I20260619 14:37:55.658074 7863 raft_consensus.cc:2781] T e84fab0f0799437e8ba2f84532c2cd03 P 24dc9c1322cb4b5092de8d6a4c742d77 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:37:55.658265 7863 raft_consensus.cc:686] T e84fab0f0799437e8ba2f84532c2cd03 P 24dc9c1322cb4b5092de8d6a4c742d77 [term 1 LEADER]: Becoming Leader. State: Replica: 24dc9c1322cb4b5092de8d6a4c742d77, State: Running, Role: LEADER
I20260619 14:37:55.658510 7863 consensus_queue.cc:237] T e84fab0f0799437e8ba2f84532c2cd03 P 24dc9c1322cb4b5092de8d6a4c742d77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77032b2db22744feb997087f373b1209" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36403 } } peers { permanent_uuid: "24dc9c1322cb4b5092de8d6a4c742d77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46221 } } peers { permanent_uuid: "57ffea6d72ae469491ce019f5e03a885" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33487 } }
I20260619 14:37:55.658905 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:55.658684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46665 (local address 127.0.98.2:52571)
0619 14:37:55.658788 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:55.658837 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260619 14:37:55.660945 7301 catalog_manager.cc:5495] T e84fab0f0799437e8ba2f84532c2cd03 P 24dc9c1322cb4b5092de8d6a4c742d77 reported cstate change: term changed from 0 to 1, leader changed from <none> to 24dc9c1322cb4b5092de8d6a4c742d77 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "24dc9c1322cb4b5092de8d6a4c742d77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77032b2db22744feb997087f373b1209" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24dc9c1322cb4b5092de8d6a4c742d77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57ffea6d72ae469491ce019f5e03a885" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33487 } health_report { overall_health: UNKNOWN } } }
I20260619 14:37:55.678982 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:55.675948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49868 (local address 127.0.98.2:46221)
0619 14:37:55.676058 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:55.676061 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:55.676074 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:37:55.676162 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:55.676164 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:55.676214 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:55.676296 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:37:55.676470 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:55.676983 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:55.677727 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:55.677934 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:55.677998 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:55.678207 (+ 209us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:37:55.678615 (+ 408us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:37:55.678694 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:55.678813 (+ 119us) server_negotiation.cc:299] Negotiation successful
0619 14:37:55.678859 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 14:37:55.681423 7708 raft_consensus.cc:1261] T e84fab0f0799437e8ba2f84532c2cd03 P 77032b2db22744feb997087f373b1209 [term 1 FOLLOWER]: Refusing update from remote peer 24dc9c1322cb4b5092de8d6a4c742d77: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:37:55.681689 7435 raft_consensus.cc:1261] T e84fab0f0799437e8ba2f84532c2cd03 P 57ffea6d72ae469491ce019f5e03a885 [term 1 FOLLOWER]: Refusing update from remote peer 24dc9c1322cb4b5092de8d6a4c742d77: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:37:55.681901 7868 consensus_queue.cc:1035] T e84fab0f0799437e8ba2f84532c2cd03 P 24dc9c1322cb4b5092de8d6a4c742d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77032b2db22744feb997087f373b1209" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:37:55.682176 7863 consensus_queue.cc:1035] T e84fab0f0799437e8ba2f84532c2cd03 P 24dc9c1322cb4b5092de8d6a4c742d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57ffea6d72ae469491ce019f5e03a885" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:37:55.684531 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:55.684014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46665 (local address 127.0.98.3:60547)
0619 14:37:55.684390 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:55.684452 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303}
I20260619 14:37:55.684854 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:55.684501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46665 (local address 127.0.98.1:59997)
0619 14:37:55.684702 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:55.684786 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260619 14:37:55.686900 7874 mvcc.cc:205] Tried to move back new op lower bound from 7298579970786713600 to 7298579970700021760. Current Snapshot: MvccSnapshot[applied={T|T < 7298579970786713600}]
I20260619 14:37:55.687754 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:55.687380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46665 (local address 127.0.98.1:42895)
0619 14:37:55.687641 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:55.687691 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200}
I20260619 14:37:55.688601 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:55.687380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46665 (local address 127.0.98.2:60733)
0619 14:37:55.688483 (+ 1103us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:55.688536 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1036}
I20260619 14:37:55.689394 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:55.689058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46665 (local address 127.0.98.3:46359)
0619 14:37:55.689281 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:55.689332 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162}
I20260619 14:37:55.691207 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:55.691016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46665 (local address 127.0.98.2:49933)
0619 14:37:55.691106 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:55.691151 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260619 14:37:55.696010 7302 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:37:55.696157 7302 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:37:55.699625 7302 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 827bf26a4cd440e59c8db36bbc95d991: Sending DeleteTablet for 3 replicas of tablet e84fab0f0799437e8ba2f84532c2cd03
I20260619 14:37:55.700150 7688 tablet_service.cc:1514] Processing DeleteTablet for tablet e84fab0f0799437e8ba2f84532c2cd03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:37:55 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:34954
I20260619 14:37:55.700475 7552 tablet_service.cc:1514] Processing DeleteTablet for tablet e84fab0f0799437e8ba2f84532c2cd03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:37:55 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:49862
I20260619 14:37:55.700524 7415 tablet_service.cc:1514] Processing DeleteTablet for tablet e84fab0f0799437e8ba2f84532c2cd03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:37:55 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:43684
I20260619 14:37:55.700806 7882 tablet_replica.cc:330] T e84fab0f0799437e8ba2f84532c2cd03 P 24dc9c1322cb4b5092de8d6a4c742d77: stopping tablet replica
I20260619 14:37:55.700927 7882 raft_consensus.cc:2229] T e84fab0f0799437e8ba2f84532c2cd03 P 24dc9c1322cb4b5092de8d6a4c742d77 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:37:55.701205 7882 raft_consensus.cc:2258] T e84fab0f0799437e8ba2f84532c2cd03 P 24dc9c1322cb4b5092de8d6a4c742d77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:37:55.701612 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 7263
I20260619 14:37:55.703259 7882 ts_tablet_manager.cc:1905] T e84fab0f0799437e8ba2f84532c2cd03 P 24dc9c1322cb4b5092de8d6a4c742d77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:37:55.705826 7882 ts_tablet_manager.cc:1918] T e84fab0f0799437e8ba2f84532c2cd03 P 24dc9c1322cb4b5092de8d6a4c742d77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:37:55.705966 7882 log.cc:1196] T e84fab0f0799437e8ba2f84532c2cd03 P 24dc9c1322cb4b5092de8d6a4c742d77: Deleting WAL directory at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-1/wal/wals/e84fab0f0799437e8ba2f84532c2cd03
I20260619 14:37:55.706250 7882 ts_tablet_manager.cc:1939] T e84fab0f0799437e8ba2f84532c2cd03 P 24dc9c1322cb4b5092de8d6a4c742d77: Deleting consensus metadata
I20260619 14:37:55.706923 7287 catalog_manager.cc:4841] TS 24dc9c1322cb4b5092de8d6a4c742d77 (127.0.98.2:46221): tablet e84fab0f0799437e8ba2f84532c2cd03 (table test-table [id=186544586a75462e9475436082b11b6d]) successfully deleted
I20260619 14:37:55.708851 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:55.708575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33487 (local address 127.0.0.1:43688)
0619 14:37:55.708696 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:55.708772 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260619 14:37:55.708858 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 7489
W20260619 14:37:55.709429 7286 catalog_manager.cc:4584] TS 57ffea6d72ae469491ce019f5e03a885 (127.0.98.1:33487): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e84fab0f0799437e8ba2f84532c2cd03: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33487: connect: Connection refused (error 111)
I20260619 14:37:55.709501 7881 tablet_replica.cc:330] T e84fab0f0799437e8ba2f84532c2cd03 P 77032b2db22744feb997087f373b1209: stopping tablet replica
I20260619 14:37:55.709627 7881 raft_consensus.cc:2229] T e84fab0f0799437e8ba2f84532c2cd03 P 77032b2db22744feb997087f373b1209 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:37:55.709789 7881 raft_consensus.cc:2258] T e84fab0f0799437e8ba2f84532c2cd03 P 77032b2db22744feb997087f373b1209 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:37:55.710414 7881 ts_tablet_manager.cc:1905] T e84fab0f0799437e8ba2f84532c2cd03 P 77032b2db22744feb997087f373b1209: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:37:55.712334 7881 ts_tablet_manager.cc:1918] T e84fab0f0799437e8ba2f84532c2cd03 P 77032b2db22744feb997087f373b1209: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:37:55.712404 7881 log.cc:1196] T e84fab0f0799437e8ba2f84532c2cd03 P 77032b2db22744feb997087f373b1209: Deleting WAL directory at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781879837648036-392-0/minicluster-data/ts-2/wal/wals/e84fab0f0799437e8ba2f84532c2cd03
I20260619 14:37:55.712639 7881 ts_tablet_manager.cc:1939] T e84fab0f0799437e8ba2f84532c2cd03 P 77032b2db22744feb997087f373b1209: Deleting consensus metadata
I20260619 14:37:55.713284 7286 catalog_manager.cc:4841] TS 77032b2db22744feb997087f373b1209 (127.0.98.3:36403): tablet e84fab0f0799437e8ba2f84532c2cd03 (table test-table [id=186544586a75462e9475436082b11b6d]) successfully deleted
I20260619 14:37:55.714711 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 7625
I20260619 14:37:55.719709 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 6976
I20260619 14:37:55.724246 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 7083
2026-06-19T14:37:55Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6618 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:37:55 dist-test-slave-32hj krb5kdc[7886](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:37:55 dist-test-slave-32hj krb5kdc[7886](info): set up 2 sockets
Jun 19 14:37:55 dist-test-slave-32hj krb5kdc[7886](info): commencing operation
krb5kdc: starting...
W20260619 14:37:56.688939 392 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/krb5kdc/test-user.keytab.
Jun 19 14:37:56 dist-test-slave-32hj krb5kdc[7886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:37:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:37:56Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260619 14:37:56.799283 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44517
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44517
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:37:56.902784 7899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:37:56.903041 7899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:37:56.903079 7899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:37:56.907032 7899 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:37:56.907085 7899 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 14:37:56.907101 7899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:37:56.907117 7899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:37:56.907132 7899 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:37:56.907150 7899 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:37:56.912220 7899 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44517
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:37:56.913216 7899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:37:56.914214 7899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:37:56.922034 7907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:56.922070 7904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:56.922044 7905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:56.922271 7899 server_base.cc:1034] running on GCE node
I20260619 14:37:56.923025 7899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:37:56.924064 7899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:37:56.925220 7899 hybrid_clock.cc:648] HybridClock initialized: now 1781879876925199 us; error 29 us; skew 500 ppm
Jun 19 14:37:56 dist-test-slave-32hj krb5kdc[7886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879876, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:37:56.927912 7899 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260619 14:37:56.928851 7899 webserver.cc:460] Webserver started at http://127.0.98.62:32895/ using document root <none> and password file <none>
I20260619 14:37:56.929379 7899 fs_manager.cc:359] Metadata directory not provided
I20260619 14:37:56.929445 7899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:37:56.929649 7899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:37:56.931666 7899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/master-0/data/instance:
uuid: "d7976811491a44f7add55a6e47dc7e97"
format_stamp: "Formatted at 2026-06-19 14:37:56 on dist-test-slave-32hj"
server_key: "598185a77afd85220a3c569f620f30a5"
server_key_iv: "f7d4b6464263f62999b515b116ce6eea"
server_key_version: "encryptionkey@0"
I20260619 14:37:56.932194 7899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/master-0/wal/instance:
uuid: "d7976811491a44f7add55a6e47dc7e97"
format_stamp: "Formatted at 2026-06-19 14:37:56 on dist-test-slave-32hj"
server_key: "598185a77afd85220a3c569f620f30a5"
server_key_iv: "f7d4b6464263f62999b515b116ce6eea"
server_key_version: "encryptionkey@0"
I20260619 14:37:56.935770 7899 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260619 14:37:56.938198 7914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:56.939419 7899 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:37:56.939535 7899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/master-0/wal
uuid: "d7976811491a44f7add55a6e47dc7e97"
format_stamp: "Formatted at 2026-06-19 14:37:56 on dist-test-slave-32hj"
server_key: "598185a77afd85220a3c569f620f30a5"
server_key_iv: "f7d4b6464263f62999b515b116ce6eea"
server_key_version: "encryptionkey@0"
I20260619 14:37:56.939661 7899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:37:56.960517 7899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:37:56.963646 7899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:37:56.963807 7899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:37:56.970824 7899 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44517
I20260619 14:37:56.970885 7976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44517 every 8 connection(s)
I20260619 14:37:56.971710 7899 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:37:56.974560 7977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:56.974704 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 7899
I20260619 14:37:56.974828 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/master-0/wal/instance
I20260619 14:37:56.975077 392 external_mini_cluster.cc:1428] Setting key 73abaf8d50d7af0820167cb548251a8f
I20260619 14:37:56.980103 7977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7976811491a44f7add55a6e47dc7e97: Bootstrap starting.
I20260619 14:37:56.982263 7977 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7976811491a44f7add55a6e47dc7e97: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:56.982981 7977 log.cc:824] T 00000000000000000000000000000000 P d7976811491a44f7add55a6e47dc7e97: Log is configured to *not* fsync() on all Append() calls
Jun 19 14:37:56 dist-test-slave-32hj krb5kdc[7886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 14:37:56.984881 7977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7976811491a44f7add55a6e47dc7e97: No bootstrap required, opened a new log
I20260619 14:37:56.987738 7977 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7976811491a44f7add55a6e47dc7e97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7976811491a44f7add55a6e47dc7e97" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44517 } }
I20260619 14:37:56.988003 7977 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7976811491a44f7add55a6e47dc7e97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:56.988076 7977 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7976811491a44f7add55a6e47dc7e97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7976811491a44f7add55a6e47dc7e97, State: Initialized, Role: FOLLOWER
I20260619 14:37:56.988489 7977 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7976811491a44f7add55a6e47dc7e97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7976811491a44f7add55a6e47dc7e97" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44517 } }
I20260619 14:37:56.988618 7977 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7976811491a44f7add55a6e47dc7e97 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:37:56.988682 7977 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7976811491a44f7add55a6e47dc7e97 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:37:56.988816 7977 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7976811491a44f7add55a6e47dc7e97 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:56.989089 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:56.976268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56778 (local address 127.0.98.62:44517)
0619 14:37:56.976648 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:56.976666 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:56.976873 (+ 207us) server_negotiation.cc:407] Connection header received
0619 14:37:56.977586 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:56.977602 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:56.978002 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:56.978386 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:56.979116 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:56.980060 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:56.980945 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:56.981389 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:56.981532 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:56.984013 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:56.984018 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:56.984032 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:56.984065 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:56.986200 (+ 2135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:56.986620 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:56.986626 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:56.986630 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:56.986736 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:56.986984 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:56.986987 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:56.986989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:56.987368 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:56.987551 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:56.987813 (+ 262us) server_negotiation.cc:299] Negotiation successful
0619 14:37:56.988110 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":102,"threads_started":1}
I20260619 14:37:56.989837 7977 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7976811491a44f7add55a6e47dc7e97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7976811491a44f7add55a6e47dc7e97" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44517 } }
I20260619 14:37:56.990182 7977 leader_election.cc:302] T 00000000000000000000000000000000 P d7976811491a44f7add55a6e47dc7e97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7976811491a44f7add55a6e47dc7e97; no voters:
I20260619 14:37:56.990475 7977 leader_election.cc:288] T 00000000000000000000000000000000 P d7976811491a44f7add55a6e47dc7e97 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:37:56.990572 7982 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7976811491a44f7add55a6e47dc7e97 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:37:56.990785 7982 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7976811491a44f7add55a6e47dc7e97 [term 1 LEADER]: Becoming Leader. State: Replica: d7976811491a44f7add55a6e47dc7e97, State: Running, Role: LEADER
I20260619 14:37:56.991070 7982 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7976811491a44f7add55a6e47dc7e97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7976811491a44f7add55a6e47dc7e97" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44517 } }
I20260619 14:37:56.991602 7977 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7976811491a44f7add55a6e47dc7e97 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:37:56.992229 7984 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7976811491a44f7add55a6e47dc7e97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7976811491a44f7add55a6e47dc7e97. Latest consensus state: current_term: 1 leader_uuid: "d7976811491a44f7add55a6e47dc7e97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7976811491a44f7add55a6e47dc7e97" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44517 } } }
I20260619 14:37:56.992364 7984 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7976811491a44f7add55a6e47dc7e97 [sys.catalog]: This master's current role is: LEADER
I20260619 14:37:56.992764 7991 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:37:56.992777 7983 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7976811491a44f7add55a6e47dc7e97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7976811491a44f7add55a6e47dc7e97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7976811491a44f7add55a6e47dc7e97" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44517 } } }
I20260619 14:37:56.993005 7983 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7976811491a44f7add55a6e47dc7e97 [sys.catalog]: This master's current role is: LEADER
I20260619 14:37:56.995918 7991 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:37:57.001854 7991 catalog_manager.cc:1347] Generated new cluster ID: 35e29da6beb24f9986eb3e83244da16d
I20260619 14:37:57.001961 7991 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:37:57.009181 7991 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:37:57.010309 7991 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:37:57.027031 7991 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7976811491a44f7add55a6e47dc7e97: Generated new TSK 0
I20260619 14:37:57.027932 7991 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260619 14:37:57.063998 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44517
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:37:57.169258 8003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:37:57.169492 8003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:37:57.169549 8003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:37:57.173519 8003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:37:57.173579 8003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:37:57.173640 8003 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 14:37:57.173692 8003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:37:57.178735 8003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:37:57.179701 8003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:37:57.180663 8003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:37:57.189468 8009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:57.189525 8008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:57.189556 8011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:57.189733 8003 server_base.cc:1034] running on GCE node
I20260619 14:37:57.190191 8003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:37:57.190793 8003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:37:57.191956 8003 hybrid_clock.cc:648] HybridClock initialized: now 1781879877191926 us; error 46 us; skew 500 ppm
Jun 19 14:37:57 dist-test-slave-32hj krb5kdc[7886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879877, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:37:57.194712 8003 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260619 14:37:57.195506 8003 webserver.cc:460] Webserver started at http://127.0.98.1:42463/ using document root <none> and password file <none>
I20260619 14:37:57.196020 8003 fs_manager.cc:359] Metadata directory not provided
I20260619 14:37:57.196084 8003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:37:57.196261 8003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:37:57.198254 8003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/ts-0/data/instance:
uuid: "5a55ef951bdf498b898900f2ebe19989"
format_stamp: "Formatted at 2026-06-19 14:37:57 on dist-test-slave-32hj"
server_key: "f59fbe1e12f99d1652147be38930a0f1"
server_key_iv: "4d1444e4869f51b8a1631ed732adaa2f"
server_key_version: "encryptionkey@0"
I20260619 14:37:57.198769 8003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5a55ef951bdf498b898900f2ebe19989"
format_stamp: "Formatted at 2026-06-19 14:37:57 on dist-test-slave-32hj"
server_key: "f59fbe1e12f99d1652147be38930a0f1"
server_key_iv: "4d1444e4869f51b8a1631ed732adaa2f"
server_key_version: "encryptionkey@0"
I20260619 14:37:57.202178 8003 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:37:57.204350 8018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:57.205508 8003 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:37:57.205608 8003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/ts-0/wal
uuid: "5a55ef951bdf498b898900f2ebe19989"
format_stamp: "Formatted at 2026-06-19 14:37:57 on dist-test-slave-32hj"
server_key: "f59fbe1e12f99d1652147be38930a0f1"
server_key_iv: "4d1444e4869f51b8a1631ed732adaa2f"
server_key_version: "encryptionkey@0"
I20260619 14:37:57.205713 8003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:37:57.217525 8003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:37:57.220412 8003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:37:57.220597 8003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:37:57.222028 8003 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:37:57.222086 8003 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:57.222139 8003 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:37:57.222167 8003 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 14:37:57 dist-test-slave-32hj krb5kdc[7886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879877, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 14:37:57.232055 8003 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34993
I20260619 14:37:57.232160 8133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34993 every 8 connection(s)
I20260619 14:37:57.233039 8003 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:37:57.235327 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:57.222802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38673 (local address 127.0.98.62:44517)
0619 14:37:57.222917 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:57.222922 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:57.223967 (+ 1045us) server_negotiation.cc:407] Connection header received
0619 14:37:57.224991 (+ 1024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:57.224995 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:57.225059 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:57.225136 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:57.226307 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:57.227131 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:57.228841 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:57.229069 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:57.229166 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:57.231923 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:57.231926 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:57.231929 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:57.231955 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:57.233490 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:57.234057 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:57.234060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:57.234061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:57.234163 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:57.234441 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:57.234443 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:57.234444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:57.234616 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:57.234694 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:57.235108 (+ 414us) server_negotiation.cc:299] Negotiation successful
0619 14:37:57.235209 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 14:37:57.236053 8027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:57.223331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44517 (local address 127.0.98.1:38673)
0619 14:37:57.223811 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:57.223852 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:57.224745 (+ 893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:57.225306 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:57.225321 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:57.225666 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:57.226146 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:57.226165 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:57.227270 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:57.227274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:57.228678 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:57.228689 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:57.229211 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:57.229216 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:57.229405 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:57.229939 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:57.229959 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:57.231734 (+ 1775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:57.233614 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:57.233619 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:57.233633 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:57.233950 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:57.234253 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:57.234255 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:57.234256 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:57.234373 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:57.234686 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:57.234691 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:57.234919 (+ 228us) client_negotiation.cc:769] Sending connection context
0619 14:37:57.235069 (+ 150us) client_negotiation.cc:240] Negotiation successful
0619 14:37:57.235306 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":132,"threads_started":1}
I20260619 14:37:57.237323 8134 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44517
I20260619 14:37:57.237462 8134 heartbeater.cc:464] Registering TS with master...
I20260619 14:37:57.237939 8134 heartbeater.cc:511] Master 127.0.98.62:44517 requested a full tablet report, sending...
I20260619 14:37:57.239459 7930 ts_manager.cc:194] Registered new tserver with Master: 5a55ef951bdf498b898900f2ebe19989 (127.0.98.1:34993)
I20260619 14:37:57.239888 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 8003
I20260619 14:37:57.240018 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:37:57.240294 392 external_mini_cluster.cc:1428] Setting key dfb5943438d3b73c783e51c9a31a8adb
I20260619 14:37:57.240932 7930 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38673
I20260619 14:37:57.246467 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:57.239218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60989 (local address 127.0.98.62:44517)
0619 14:37:57.239327 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:57.239331 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:57.239428 (+ 97us) server_negotiation.cc:407] Connection header received
0619 14:37:57.239512 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:57.239514 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:57.239565 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:57.239647 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:57.240128 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:57.240943 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:57.241992 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:57.242311 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:57.242407 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:57.243631 (+ 1224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:57.243635 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:57.243638 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:57.243662 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:57.245033 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:57.245456 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:57.245459 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:57.245460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:57.245547 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:57.245800 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:57.245802 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:57.245803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:57.245952 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:57.246018 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:57.246274 (+ 256us) server_negotiation.cc:299] Negotiation successful
0619 14:37:57.246358 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":11,"server-negotiator.queue_time_us":47}
I20260619 14:37:57.246569 8140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:57.239114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44517 (local address 127.0.98.1:60989)
0619 14:37:57.239355 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:57.239367 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:57.239442 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:57.239674 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:57.239677 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:57.239873 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:57.239995 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:57.240002 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:57.241068 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:57.241071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:57.241858 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:57.241869 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:57.242533 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:57.242537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:57.242625 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:57.243089 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:57.243102 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:57.243497 (+ 395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:57.245127 (+ 1630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:57.245131 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:57.245133 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:57.245368 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:57.245633 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:57.245636 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:57.245638 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:57.245727 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:57.246033 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:57.246036 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:57.246140 (+ 104us) client_negotiation.cc:769] Sending connection context
0619 14:37:57.246247 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 14:37:57.246381 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":86,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260619 14:37:57.264271 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44517
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:37:57.371526 8143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:37:57.371788 8143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:37:57.371832 8143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:37:57.375759 8143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:37:57.375821 8143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:37:57.375861 8143 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 14:37:57.375895 8143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:37:57.381103 8143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:37:57.382091 8143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:37:57.383059 8143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:37:57.392026 8143 server_base.cc:1034] running on GCE node
W20260619 14:37:57.392009 8148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:57.392004 8149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:57.392309 8151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:57.392657 8143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:37:57.393321 8143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:37:57.394470 8143 hybrid_clock.cc:648] HybridClock initialized: now 1781879877394455 us; error 33 us; skew 500 ppm
Jun 19 14:37:57 dist-test-slave-32hj krb5kdc[7886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879877, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:37:57.396981 8143 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260619 14:37:57.397786 8143 webserver.cc:460] Webserver started at http://127.0.98.2:34471/ using document root <none> and password file <none>
I20260619 14:37:57.398284 8143 fs_manager.cc:359] Metadata directory not provided
I20260619 14:37:57.398348 8143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:37:57.398520 8143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:37:57.400467 8143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/ts-1/data/instance:
uuid: "85b530590f0848d99c9df9ee4a75e71c"
format_stamp: "Formatted at 2026-06-19 14:37:57 on dist-test-slave-32hj"
server_key: "62e6265b8bb9f36f9a81860a143bf6cc"
server_key_iv: "caf02b5e2409c2784253cc5c0fc306a9"
server_key_version: "encryptionkey@0"
I20260619 14:37:57.400987 8143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/ts-1/wal/instance:
uuid: "85b530590f0848d99c9df9ee4a75e71c"
format_stamp: "Formatted at 2026-06-19 14:37:57 on dist-test-slave-32hj"
server_key: "62e6265b8bb9f36f9a81860a143bf6cc"
server_key_iv: "caf02b5e2409c2784253cc5c0fc306a9"
server_key_version: "encryptionkey@0"
I20260619 14:37:57.404371 8143 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 14:37:57.406543 8158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:57.407631 8143 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 14:37:57.407729 8143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/ts-1/wal
uuid: "85b530590f0848d99c9df9ee4a75e71c"
format_stamp: "Formatted at 2026-06-19 14:37:57 on dist-test-slave-32hj"
server_key: "62e6265b8bb9f36f9a81860a143bf6cc"
server_key_iv: "caf02b5e2409c2784253cc5c0fc306a9"
server_key_version: "encryptionkey@0"
I20260619 14:37:57.407817 8143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:37:57.437989 8143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:37:57.441112 8143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:37:57.441294 8143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:37:57.442683 8143 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:37:57.442732 8143 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:57.442785 8143 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:37:57.442811 8143 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 14:37:57 dist-test-slave-32hj krb5kdc[7886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879877, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 14:37:57.452960 8143 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41193
I20260619 14:37:57.453114 8273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41193 every 8 connection(s)
I20260619 14:37:57.453827 8143 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:37:57.455727 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:57.443603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36701 (local address 127.0.98.62:44517)
0619 14:37:57.443713 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:57.443717 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:57.444776 (+ 1059us) server_negotiation.cc:407] Connection header received
0619 14:37:57.445819 (+ 1043us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:57.445823 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:57.445881 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:57.445963 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:57.447173 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:57.447983 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:57.449431 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:57.449803 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:57.449925 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:57.452391 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:57.452394 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:57.452396 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:57.452420 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:57.453804 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:57.454356 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:57.454360 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:57.454362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:57.454456 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:57.454790 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:57.454793 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:57.454794 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:57.454974 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:57.455059 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:57.455512 (+ 453us) server_negotiation.cc:299] Negotiation successful
0619 14:37:57.455615 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 14:37:57.456465 8167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:57.444157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44517 (local address 127.0.98.2:36701)
0619 14:37:57.444622 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:57.444665 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:57.445594 (+ 929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:57.446145 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:57.446152 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:57.446491 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:57.447019 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:57.447038 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:57.448114 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:57.448118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:57.449329 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:57.449338 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:57.449968 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:57.449972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:57.450169 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:57.450710 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:57.450726 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:57.452224 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:57.453910 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:57.453915 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:57.453929 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:57.454244 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:57.454550 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:57.454553 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:57.454555 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:57.454675 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:57.455074 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:57.455078 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:57.455292 (+ 214us) client_negotiation.cc:769] Sending connection context
0619 14:37:57.455439 (+ 147us) client_negotiation.cc:240] Negotiation successful
0619 14:37:57.455663 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":131,"threads_started":1}
I20260619 14:37:57.457652 8274 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44517
I20260619 14:37:57.457806 8274 heartbeater.cc:464] Registering TS with master...
I20260619 14:37:57.458251 8274 heartbeater.cc:511] Master 127.0.98.62:44517 requested a full tablet report, sending...
I20260619 14:37:57.459152 7931 ts_manager.cc:194] Registered new tserver with Master: 85b530590f0848d99c9df9ee4a75e71c (127.0.98.2:41193)
I20260619 14:37:57.459816 7931 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36701
I20260619 14:37:57.460311 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 8143
I20260619 14:37:57.460436 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:37:57.460811 392 external_mini_cluster.cc:1428] Setting key 48cc0c71a193d945b0abac203e11dce6
I20260619 14:37:57.465865 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:57.459555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45071 (local address 127.0.98.62:44517)
0619 14:37:57.459660 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:57.459664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:57.459785 (+ 121us) server_negotiation.cc:407] Connection header received
0619 14:37:57.459909 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:57.459912 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:57.459954 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:57.460014 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:57.460415 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:57.461061 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:57.461895 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:57.462148 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:57.462244 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:57.463185 (+ 941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:57.463188 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:57.463191 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:57.463215 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:57.464459 (+ 1244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:57.464874 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:57.464876 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:57.464877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:57.464940 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:57.465162 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:57.465164 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:57.465165 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:57.465331 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:57.465397 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:57.465625 (+ 228us) server_negotiation.cc:299] Negotiation successful
0619 14:37:57.465703 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 14:37:57.466105 8280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:57.459461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44517 (local address 127.0.98.2:45071)
0619 14:37:57.459708 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:57.459722 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:57.459815 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:57.460051 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:57.460054 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:57.460216 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:57.460304 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:57.460309 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:57.461170 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:57.461173 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:57.461785 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:57.461794 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:57.462255 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:57.462258 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:57.462327 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:57.462771 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:57.462781 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:57.463053 (+ 272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:57.464556 (+ 1503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:57.464560 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:57.464562 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:57.464791 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:57.465008 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:57.465011 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:57.465013 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:57.465093 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:57.465403 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:57.465405 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:57.465489 (+ 84us) client_negotiation.cc:769] Sending connection context
0619 14:37:57.465592 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 14:37:57.465730 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":85,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260619 14:37:57.483655 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44517
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:37:57.587165 8283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:37:57.587399 8283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:37:57.587442 8283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:37:57.591307 8283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:37:57.591364 8283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:37:57.591415 8283 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 14:37:57.591470 8283 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:37:57.596529 8283 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8283
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:37:57.597455 8283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:37:57.598362 8283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:37:57.607182 8289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:37:57.607312 8288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:57.607354 8283 server_base.cc:1034] running on GCE node
W20260619 14:37:57.607191 8291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:37:57.608023 8283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:37:57.608696 8283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:37:57.609881 8283 hybrid_clock.cc:648] HybridClock initialized: now 1781879877609865 us; error 32 us; skew 500 ppm
Jun 19 14:37:57 dist-test-slave-32hj krb5kdc[7886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879877, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:37:57.612581 8283 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260619 14:37:57.613420 8283 webserver.cc:460] Webserver started at http://127.0.98.3:39221/ using document root <none> and password file <none>
I20260619 14:37:57.613940 8283 fs_manager.cc:359] Metadata directory not provided
I20260619 14:37:57.614001 8283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:37:57.614176 8283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:37:57.616108 8283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/ts-2/data/instance:
uuid: "9f164e7dbb0347039e10be0c23084924"
format_stamp: "Formatted at 2026-06-19 14:37:57 on dist-test-slave-32hj"
server_key: "f08a1735206afd60c3f842417e2fdf76"
server_key_iv: "24383058c4c5350f8da966aa2fe5c66f"
server_key_version: "encryptionkey@0"
I20260619 14:37:57.616626 8283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9f164e7dbb0347039e10be0c23084924"
format_stamp: "Formatted at 2026-06-19 14:37:57 on dist-test-slave-32hj"
server_key: "f08a1735206afd60c3f842417e2fdf76"
server_key_iv: "24383058c4c5350f8da966aa2fe5c66f"
server_key_version: "encryptionkey@0"
I20260619 14:37:57.620134 8283 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260619 14:37:57.622395 8298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:57.623585 8283 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:37:57.623701 8283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/ts-2/wal
uuid: "9f164e7dbb0347039e10be0c23084924"
format_stamp: "Formatted at 2026-06-19 14:37:57 on dist-test-slave-32hj"
server_key: "f08a1735206afd60c3f842417e2fdf76"
server_key_iv: "24383058c4c5350f8da966aa2fe5c66f"
server_key_version: "encryptionkey@0"
I20260619 14:37:57.623801 8283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:37:57.643311 8283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:37:57.646258 8283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:37:57.646423 8283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:37:57.647784 8283 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:37:57.647832 8283 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:37:57.647883 8283 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:37:57.647912 8283 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 14:37:57 dist-test-slave-32hj krb5kdc[7886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879877, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 14:37:57.657673 8283 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39321
I20260619 14:37:57.657799 8413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39321 every 8 connection(s)
I20260619 14:37:57.658545 8283 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:37:57.658771 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 8283
I20260619 14:37:57.658879 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:37:57.659171 392 external_mini_cluster.cc:1428] Setting key daa03d1f0a40d74ae9d2686b5405f55c
I20260619 14:37:57.661410 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:57.648525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52721 (local address 127.0.98.62:44517)
0619 14:37:57.648634 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:57.648637 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:57.649682 (+ 1045us) server_negotiation.cc:407] Connection header received
0619 14:37:57.650661 (+ 979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:57.650664 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:57.650745 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:57.650805 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:57.652158 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:57.652807 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:57.654491 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:57.654834 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:57.654939 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:57.657634 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:57.657637 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:57.657640 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:57.657667 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:57.659210 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:57.659760 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:57.659764 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:57.659766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:57.659861 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:57.660194 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:57.660197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:57.660199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:57.660414 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:57.660502 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:57.661155 (+ 653us) server_negotiation.cc:299] Negotiation successful
0619 14:37:57.661266 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 14:37:57.662052 8307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:57.649099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44517 (local address 127.0.98.3:52721)
0619 14:37:57.649547 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:57.649587 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:57.650422 (+ 835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:57.651211 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:57.651220 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:57.651527 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:57.652025 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:57.652039 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:57.652948 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:57.652952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:57.654360 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:57.654372 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:57.654988 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:57.654992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:57.655184 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:57.655723 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:57.655744 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:57.657484 (+ 1740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:57.659328 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:57.659333 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:57.659347 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:57.659652 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:57.659940 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:57.659943 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:57.659945 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:57.660111 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:57.660512 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:57.660518 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:57.660795 (+ 277us) client_negotiation.cc:769] Sending connection context
0619 14:37:57.660961 (+ 166us) client_negotiation.cc:240] Negotiation successful
0619 14:37:57.661264 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":132,"threads_started":1}
I20260619 14:37:57.663075 8414 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44517
I20260619 14:37:57.663218 8414 heartbeater.cc:464] Registering TS with master...
I20260619 14:37:57.663652 8414 heartbeater.cc:511] Master 127.0.98.62:44517 requested a full tablet report, sending...
I20260619 14:37:57.664593 7930 ts_manager.cc:194] Registered new tserver with Master: 9f164e7dbb0347039e10be0c23084924 (127.0.98.3:39321)
I20260619 14:37:57.665319 7930 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52721
I20260619 14:37:57.670895 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:57.665024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53185 (local address 127.0.98.62:44517)
0619 14:37:57.665131 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:57.665135 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:57.665260 (+ 125us) server_negotiation.cc:407] Connection header received
0619 14:37:57.665365 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:57.665367 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:57.665404 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:57.665480 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:57.665912 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:57.666403 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:57.667173 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:57.667381 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:57.667460 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:57.668385 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:57.668388 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:57.668390 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:57.668408 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:57.669553 (+ 1145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:57.669911 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:57.669915 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:57.669916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:57.669981 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:57.670215 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:57.670218 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:57.670220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:57.670366 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:57.670444 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:57.670669 (+ 225us) server_negotiation.cc:299] Negotiation successful
0619 14:37:57.670761 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 14:37:57.670907 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:57.664926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44517 (local address 127.0.98.3:53185)
0619 14:37:57.665184 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:57.665199 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:57.665290 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:57.665502 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:57.665505 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:57.665710 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:57.665822 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:57.665829 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:57.666510 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:57.666514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:57.667074 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:57.667081 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:57.667470 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:57.667473 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:57.667556 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:57.667976 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:57.667990 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:57.668265 (+ 275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:57.669643 (+ 1378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:57.669646 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:57.669648 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:57.669830 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:57.670056 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:57.670059 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:57.670061 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:57.670135 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:57.670447 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:57.670450 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:57.670532 (+ 82us) client_negotiation.cc:769] Sending connection context
0619 14:37:57.670612 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 14:37:57.670741 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":93,"threads_started":1}
I20260619 14:37:57.672644 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 14:37:57 dist-test-slave-32hj krb5kdc[7886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879877, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 14:37:57 dist-test-slave-32hj krb5kdc[7886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:37:57 dist-test-slave-32hj krb5kdc[7886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 14:37:57.694137 7980 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:37:57.688338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56792 (local address 127.0.98.62:44517)
0619 14:37:57.688449 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:57.688452 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:57.688465 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:37:57.688506 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:57.688509 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:57.688548 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:57.688614 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:57.689282 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:57.689796 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:57.690601 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:57.690818 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:57.690888 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:57.693992 (+ 3104us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56792: BlockingRecv error: recv got EOF from 127.0.0.1:56792 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Jun 19 14:37:57 dist-test-slave-32hj krb5kdc[7886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879877, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 14:37:57.702508 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:57.695526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56794 (local address 127.0.98.62:44517)
0619 14:37:57.695646 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:57.695650 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:57.695740 (+ 90us) server_negotiation.cc:407] Connection header received
0619 14:37:57.695844 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:57.695847 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:57.695886 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:57.695947 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:57.696498 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:57.697082 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:57.697813 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:57.698014 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:57.698092 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:57.700021 (+ 1929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:57.700023 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:57.700025 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:57.700046 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:57.701189 (+ 1143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:57.701537 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:57.701540 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:57.701541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:57.701602 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:57.701845 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:57.701847 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:57.701848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:57.701992 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:57.702084 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:57.702313 (+ 229us) server_negotiation.cc:299] Negotiation successful
0619 14:37:57.702397 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 14:37:57.705206 7931 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56794:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:37:57.708015 7931 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:37:57.721397 8436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:57.715478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39321 (local address 127.0.0.1:58214)
0619 14:37:57.715823 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:57.715873 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:57.716325 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:57.717088 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:57.717094 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:57.717116 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:37:57.717222 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:57.717227 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:57.718328 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:57.718331 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:57.719941 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:57.719952 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:57.720851 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:57.720855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:57.720975 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:57.721020 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 14:37:57.721119 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 14:37:57.721254 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":92,"thread_start_us":78,"threads_started":1}
I20260619 14:37:57.721400 8437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:57.715587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58214 (local address 127.0.98.3:39321)
0619 14:37:57.716017 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:57.716032 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:57.716056 (+ 24us) server_negotiation.cc:407] Connection header received
0619 14:37:57.716528 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:57.716534 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:57.716810 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:57.716945 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:37:57.717331 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:57.718244 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:57.720058 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:57.720708 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:57.720844 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:57.721009 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:57.721154 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 14:37:57.721249 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":111,"threads_started":1}
I20260619 14:37:57.721560 8439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:57.716145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41193 (local address 127.0.0.1:50132)
0619 14:37:57.717420 (+ 1275us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:57.717431 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:57.717514 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:57.717912 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:57.717915 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:57.717932 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:37:57.718023 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:57.718028 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:57.719125 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:57.719128 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:57.720198 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:57.720217 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:57.721221 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:57.721224 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:57.721310 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:57.721331 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 14:37:57.721394 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 14:37:57.721454 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1218,"mutex_wait_us":165,"thread_start_us":87,"threads_started":1}
I20260619 14:37:57.721937 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:57.716517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50132 (local address 127.0.98.2:41193)
0619 14:37:57.716971 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:57.716983 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:57.717511 (+ 528us) server_negotiation.cc:407] Connection header received
0619 14:37:57.717608 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:57.717613 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:57.717785 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:57.717924 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:37:57.718121 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:57.719020 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:57.720316 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:57.720925 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:57.721020 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:57.721165 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:57.721704 (+ 539us) server_negotiation.cc:299] Negotiation successful
0619 14:37:57.721809 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":111,"threads_started":1}
I20260619 14:37:57.722306 8438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:57.715811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34993 (local address 127.0.0.1:60504)
0619 14:37:57.716492 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:57.716505 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:57.716575 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:57.717246 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:57.717249 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:57.717266 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:37:57.717364 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:57.717370 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:57.718652 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:57.718655 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:57.719952 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:57.719959 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:57.721942 (+ 1983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:57.721946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:57.722043 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:57.722065 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 14:37:57.722124 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 14:37:57.722189 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"mutex_wait_us":118,"thread_start_us":81,"threads_started":1}
I20260619 14:37:57.723912 8440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:57.716226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60504 (local address 127.0.98.1:34993)
0619 14:37:57.716897 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:57.716910 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:57.716929 (+ 19us) server_negotiation.cc:407] Connection header received
0619 14:37:57.716985 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:57.716988 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:57.717116 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:57.717246 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:37:57.717463 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:57.718538 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:57.720806 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:57.721838 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:57.721933 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:57.723523 (+ 1590us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:57.723573 (+ 50us) server_negotiation.cc:299] Negotiation successful
0619 14:37:57.723662 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":84,"threads_started":1}
I20260619 14:37:57.724259 8348 tablet_service.cc:1467] Processing CreateTablet for tablet 556f712670934445b08946c1a5272461 (DEFAULT_TABLE table=test-table [id=36488bad5cc54b29892b867120d788b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:37:57.724795 8208 tablet_service.cc:1467] Processing CreateTablet for tablet 556f712670934445b08946c1a5272461 (DEFAULT_TABLE table=test-table [id=36488bad5cc54b29892b867120d788b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:37:57.725221 8348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 556f712670934445b08946c1a5272461. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:57.725791 8208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 556f712670934445b08946c1a5272461. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:57.726440 8068 tablet_service.cc:1467] Processing CreateTablet for tablet 556f712670934445b08946c1a5272461 (DEFAULT_TABLE table=test-table [id=36488bad5cc54b29892b867120d788b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:37:57.727432 8068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 556f712670934445b08946c1a5272461. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:57.731459 8442 tablet_bootstrap.cc:492] T 556f712670934445b08946c1a5272461 P 9f164e7dbb0347039e10be0c23084924: Bootstrap starting.
I20260619 14:37:57.733239 8443 tablet_bootstrap.cc:492] T 556f712670934445b08946c1a5272461 P 85b530590f0848d99c9df9ee4a75e71c: Bootstrap starting.
I20260619 14:37:57.733917 8442 tablet_bootstrap.cc:654] T 556f712670934445b08946c1a5272461 P 9f164e7dbb0347039e10be0c23084924: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:57.734916 8442 log.cc:824] T 556f712670934445b08946c1a5272461 P 9f164e7dbb0347039e10be0c23084924: Log is configured to *not* fsync() on all Append() calls
I20260619 14:37:57.735287 8443 tablet_bootstrap.cc:654] T 556f712670934445b08946c1a5272461 P 85b530590f0848d99c9df9ee4a75e71c: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:57.735458 8444 tablet_bootstrap.cc:492] T 556f712670934445b08946c1a5272461 P 5a55ef951bdf498b898900f2ebe19989: Bootstrap starting.
I20260619 14:37:57.736078 8443 log.cc:824] T 556f712670934445b08946c1a5272461 P 85b530590f0848d99c9df9ee4a75e71c: Log is configured to *not* fsync() on all Append() calls
I20260619 14:37:57.736893 8442 tablet_bootstrap.cc:492] T 556f712670934445b08946c1a5272461 P 9f164e7dbb0347039e10be0c23084924: No bootstrap required, opened a new log
I20260619 14:37:57.737078 8442 ts_tablet_manager.cc:1397] T 556f712670934445b08946c1a5272461 P 9f164e7dbb0347039e10be0c23084924: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 14:37:57.737380 8444 tablet_bootstrap.cc:654] T 556f712670934445b08946c1a5272461 P 5a55ef951bdf498b898900f2ebe19989: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:57.738157 8444 log.cc:824] T 556f712670934445b08946c1a5272461 P 5a55ef951bdf498b898900f2ebe19989: Log is configured to *not* fsync() on all Append() calls
I20260619 14:37:57.738355 8443 tablet_bootstrap.cc:492] T 556f712670934445b08946c1a5272461 P 85b530590f0848d99c9df9ee4a75e71c: No bootstrap required, opened a new log
I20260619 14:37:57.738515 8443 ts_tablet_manager.cc:1397] T 556f712670934445b08946c1a5272461 P 85b530590f0848d99c9df9ee4a75e71c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:37:57.739929 8444 tablet_bootstrap.cc:492] T 556f712670934445b08946c1a5272461 P 5a55ef951bdf498b898900f2ebe19989: No bootstrap required, opened a new log
I20260619 14:37:57.740085 8444 ts_tablet_manager.cc:1397] T 556f712670934445b08946c1a5272461 P 5a55ef951bdf498b898900f2ebe19989: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 14:37:57.740159 8442 raft_consensus.cc:348] T 556f712670934445b08946c1a5272461 P 9f164e7dbb0347039e10be0c23084924 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f164e7dbb0347039e10be0c23084924" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39321 } } peers { permanent_uuid: "85b530590f0848d99c9df9ee4a75e71c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41193 } } peers { permanent_uuid: "5a55ef951bdf498b898900f2ebe19989" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } }
I20260619 14:37:57.740393 8442 raft_consensus.cc:374] T 556f712670934445b08946c1a5272461 P 9f164e7dbb0347039e10be0c23084924 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:57.740437 8442 raft_consensus.cc:729] T 556f712670934445b08946c1a5272461 P 9f164e7dbb0347039e10be0c23084924 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f164e7dbb0347039e10be0c23084924, State: Initialized, Role: FOLLOWER
I20260619 14:37:57.740816 8442 consensus_queue.cc:260] T 556f712670934445b08946c1a5272461 P 9f164e7dbb0347039e10be0c23084924 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f164e7dbb0347039e10be0c23084924" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39321 } } peers { permanent_uuid: "85b530590f0848d99c9df9ee4a75e71c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41193 } } peers { permanent_uuid: "5a55ef951bdf498b898900f2ebe19989" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } }
I20260619 14:37:57.741377 8414 heartbeater.cc:503] Master 127.0.98.62:44517 was elected leader, sending a full tablet report...
I20260619 14:37:57.741703 8442 ts_tablet_manager.cc:1428] T 556f712670934445b08946c1a5272461 P 9f164e7dbb0347039e10be0c23084924: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:37:57.742115 8443 raft_consensus.cc:348] T 556f712670934445b08946c1a5272461 P 85b530590f0848d99c9df9ee4a75e71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f164e7dbb0347039e10be0c23084924" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39321 } } peers { permanent_uuid: "85b530590f0848d99c9df9ee4a75e71c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41193 } } peers { permanent_uuid: "5a55ef951bdf498b898900f2ebe19989" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } }
I20260619 14:37:57.742367 8443 raft_consensus.cc:374] T 556f712670934445b08946c1a5272461 P 85b530590f0848d99c9df9ee4a75e71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:57.742419 8443 raft_consensus.cc:729] T 556f712670934445b08946c1a5272461 P 85b530590f0848d99c9df9ee4a75e71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85b530590f0848d99c9df9ee4a75e71c, State: Initialized, Role: FOLLOWER
I20260619 14:37:57.742868 8443 consensus_queue.cc:260] T 556f712670934445b08946c1a5272461 P 85b530590f0848d99c9df9ee4a75e71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f164e7dbb0347039e10be0c23084924" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39321 } } peers { permanent_uuid: "85b530590f0848d99c9df9ee4a75e71c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41193 } } peers { permanent_uuid: "5a55ef951bdf498b898900f2ebe19989" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } }
I20260619 14:37:57.743592 8274 heartbeater.cc:503] Master 127.0.98.62:44517 was elected leader, sending a full tablet report...
I20260619 14:37:57.743836 8443 ts_tablet_manager.cc:1428] T 556f712670934445b08946c1a5272461 P 85b530590f0848d99c9df9ee4a75e71c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 14:37:57.743762 8444 raft_consensus.cc:348] T 556f712670934445b08946c1a5272461 P 5a55ef951bdf498b898900f2ebe19989 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f164e7dbb0347039e10be0c23084924" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39321 } } peers { permanent_uuid: "85b530590f0848d99c9df9ee4a75e71c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41193 } } peers { permanent_uuid: "5a55ef951bdf498b898900f2ebe19989" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } }
I20260619 14:37:57.743985 8444 raft_consensus.cc:374] T 556f712670934445b08946c1a5272461 P 5a55ef951bdf498b898900f2ebe19989 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:57.744027 8444 raft_consensus.cc:729] T 556f712670934445b08946c1a5272461 P 5a55ef951bdf498b898900f2ebe19989 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a55ef951bdf498b898900f2ebe19989, State: Initialized, Role: FOLLOWER
I20260619 14:37:57.744429 8444 consensus_queue.cc:260] T 556f712670934445b08946c1a5272461 P 5a55ef951bdf498b898900f2ebe19989 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f164e7dbb0347039e10be0c23084924" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39321 } } peers { permanent_uuid: "85b530590f0848d99c9df9ee4a75e71c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41193 } } peers { permanent_uuid: "5a55ef951bdf498b898900f2ebe19989" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } }
I20260619 14:37:57.745023 8134 heartbeater.cc:503] Master 127.0.98.62:44517 was elected leader, sending a full tablet report...
I20260619 14:37:57.745396 8444 ts_tablet_manager.cc:1428] T 556f712670934445b08946c1a5272461 P 5a55ef951bdf498b898900f2ebe19989: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260619 14:37:57.910079 8415 tablet.cc:2369] T 556f712670934445b08946c1a5272461 P 9f164e7dbb0347039e10be0c23084924: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:37:57.934901 8450 raft_consensus.cc:482] T 556f712670934445b08946c1a5272461 P 5a55ef951bdf498b898900f2ebe19989 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:37:57.935156 8450 raft_consensus.cc:504] T 556f712670934445b08946c1a5272461 P 5a55ef951bdf498b898900f2ebe19989 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f164e7dbb0347039e10be0c23084924" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39321 } } peers { permanent_uuid: "85b530590f0848d99c9df9ee4a75e71c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41193 } } peers { permanent_uuid: "5a55ef951bdf498b898900f2ebe19989" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } }
I20260619 14:37:57.936368 8450 leader_election.cc:288] T 556f712670934445b08946c1a5272461 P 5a55ef951bdf498b898900f2ebe19989 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f164e7dbb0347039e10be0c23084924 (127.0.98.3:39321), 85b530590f0848d99c9df9ee4a75e71c (127.0.98.2:41193)
I20260619 14:37:57.940053 8437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:57.936776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46377 (local address 127.0.98.3:39321)
0619 14:37:57.936888 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:57.936891 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:57.937098 (+ 207us) server_negotiation.cc:407] Connection header received
0619 14:37:57.937161 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:57.937164 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:57.937218 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:57.937307 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:37:57.937583 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:57.938103 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:57.939165 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:57.939550 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:57.939650 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:57.939713 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:57.939871 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 14:37:57.939937 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 14:37:57.940068 8451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:57.936637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39321 (local address 127.0.98.1:46377)
0619 14:37:57.936924 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:57.936938 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:57.937052 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:57.937329 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:57.937332 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:57.937353 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:37:57.937476 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:57.937482 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:57.938206 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:57.938210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:57.939041 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:57.939049 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:57.939666 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:57.939669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:57.939759 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:57.939781 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 14:37:57.939848 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 14:37:57.939916 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":120,"threads_started":1}
I20260619 14:37:57.940258 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:57.937190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59811 (local address 127.0.98.2:41193)
0619 14:37:57.937327 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:57.937331 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:57.937440 (+ 109us) server_negotiation.cc:407] Connection header received
0619 14:37:57.937546 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:57.937550 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:57.937590 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:57.937648 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:37:57.937891 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:57.938408 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:57.939378 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:57.939768 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:57.939856 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:57.939913 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:57.940081 (+ 168us) server_negotiation.cc:299] Negotiation successful
0619 14:37:57.940147 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 14:37:57.940272 8452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:57.937041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41193 (local address 127.0.98.1:59811)
0619 14:37:57.937323 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:57.937364 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:57.937455 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:57.937665 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:57.937668 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:57.937682 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:37:57.937797 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:57.937801 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:57.938523 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:57.938527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:57.939265 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:57.939275 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:57.939874 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:57.939878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:57.939979 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:57.939996 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 14:37:57.940058 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 14:37:57.940123 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":72,"threads_started":1}
I20260619 14:37:57.940919 8226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "556f712670934445b08946c1a5272461" candidate_uuid: "5a55ef951bdf498b898900f2ebe19989" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85b530590f0848d99c9df9ee4a75e71c" is_pre_election: true
I20260619 14:37:57.940930 8368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "556f712670934445b08946c1a5272461" candidate_uuid: "5a55ef951bdf498b898900f2ebe19989" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f164e7dbb0347039e10be0c23084924" is_pre_election: true
I20260619 14:37:57.941195 8226 raft_consensus.cc:2454] T 556f712670934445b08946c1a5272461 P 85b530590f0848d99c9df9ee4a75e71c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a55ef951bdf498b898900f2ebe19989 in term 0.
I20260619 14:37:57.941197 8368 raft_consensus.cc:2454] T 556f712670934445b08946c1a5272461 P 9f164e7dbb0347039e10be0c23084924 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a55ef951bdf498b898900f2ebe19989 in term 0.
I20260619 14:37:57.941637 8021 leader_election.cc:302] T 556f712670934445b08946c1a5272461 P 5a55ef951bdf498b898900f2ebe19989 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a55ef951bdf498b898900f2ebe19989, 85b530590f0848d99c9df9ee4a75e71c; no voters:
I20260619 14:37:57.941883 8450 raft_consensus.cc:2781] T 556f712670934445b08946c1a5272461 P 5a55ef951bdf498b898900f2ebe19989 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:37:57.941955 8450 raft_consensus.cc:482] T 556f712670934445b08946c1a5272461 P 5a55ef951bdf498b898900f2ebe19989 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:37:57.942003 8450 raft_consensus.cc:3037] T 556f712670934445b08946c1a5272461 P 5a55ef951bdf498b898900f2ebe19989 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:57.943044 8450 raft_consensus.cc:504] T 556f712670934445b08946c1a5272461 P 5a55ef951bdf498b898900f2ebe19989 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f164e7dbb0347039e10be0c23084924" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39321 } } peers { permanent_uuid: "85b530590f0848d99c9df9ee4a75e71c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41193 } } peers { permanent_uuid: "5a55ef951bdf498b898900f2ebe19989" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } }
I20260619 14:37:57.943385 8450 leader_election.cc:288] T 556f712670934445b08946c1a5272461 P 5a55ef951bdf498b898900f2ebe19989 [CANDIDATE]: Term 1 election: Requested vote from peers 9f164e7dbb0347039e10be0c23084924 (127.0.98.3:39321), 85b530590f0848d99c9df9ee4a75e71c (127.0.98.2:41193)
I20260619 14:37:57.943678 8368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "556f712670934445b08946c1a5272461" candidate_uuid: "5a55ef951bdf498b898900f2ebe19989" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f164e7dbb0347039e10be0c23084924"
I20260619 14:37:57.943787 8226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "556f712670934445b08946c1a5272461" candidate_uuid: "5a55ef951bdf498b898900f2ebe19989" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85b530590f0848d99c9df9ee4a75e71c"
I20260619 14:37:57.943899 8368 raft_consensus.cc:3037] T 556f712670934445b08946c1a5272461 P 9f164e7dbb0347039e10be0c23084924 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:57.943922 8226 raft_consensus.cc:3037] T 556f712670934445b08946c1a5272461 P 85b530590f0848d99c9df9ee4a75e71c [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:57.945211 8226 raft_consensus.cc:2454] T 556f712670934445b08946c1a5272461 P 85b530590f0848d99c9df9ee4a75e71c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a55ef951bdf498b898900f2ebe19989 in term 1.
I20260619 14:37:57.945684 8021 leader_election.cc:302] T 556f712670934445b08946c1a5272461 P 5a55ef951bdf498b898900f2ebe19989 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a55ef951bdf498b898900f2ebe19989, 85b530590f0848d99c9df9ee4a75e71c; no voters:
I20260619 14:37:57.945783 8368 raft_consensus.cc:2454] T 556f712670934445b08946c1a5272461 P 9f164e7dbb0347039e10be0c23084924 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a55ef951bdf498b898900f2ebe19989 in term 1.
I20260619 14:37:57.945870 8450 raft_consensus.cc:2781] T 556f712670934445b08946c1a5272461 P 5a55ef951bdf498b898900f2ebe19989 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:37:57.946074 8450 raft_consensus.cc:686] T 556f712670934445b08946c1a5272461 P 5a55ef951bdf498b898900f2ebe19989 [term 1 LEADER]: Becoming Leader. State: Replica: 5a55ef951bdf498b898900f2ebe19989, State: Running, Role: LEADER
I20260619 14:37:57.946404 8450 consensus_queue.cc:237] T 556f712670934445b08946c1a5272461 P 5a55ef951bdf498b898900f2ebe19989 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f164e7dbb0347039e10be0c23084924" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39321 } } peers { permanent_uuid: "85b530590f0848d99c9df9ee4a75e71c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41193 } } peers { permanent_uuid: "5a55ef951bdf498b898900f2ebe19989" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } }
I20260619 14:37:57.949484 7923 catalog_manager.cc:5495] T 556f712670934445b08946c1a5272461 P 5a55ef951bdf498b898900f2ebe19989 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a55ef951bdf498b898900f2ebe19989 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "5a55ef951bdf498b898900f2ebe19989" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f164e7dbb0347039e10be0c23084924" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85b530590f0848d99c9df9ee4a75e71c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a55ef951bdf498b898900f2ebe19989" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } health_report { overall_health: HEALTHY } } }
W20260619 14:37:57.954855 8275 tablet.cc:2369] T 556f712670934445b08946c1a5272461 P 85b530590f0848d99c9df9ee4a75e71c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:37:57.984423 8135 tablet.cc:2369] T 556f712670934445b08946c1a5272461 P 5a55ef951bdf498b898900f2ebe19989: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:37:58.013024 8440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:58.009656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41728 (local address 127.0.98.1:34993)
0619 14:37:58.009791 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:58.009794 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:58.009824 (+ 30us) server_negotiation.cc:407] Connection header received
0619 14:37:58.009897 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:58.009900 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:58.009954 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:58.010035 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:37:58.010244 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:58.010767 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:58.011621 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:58.011874 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:58.011953 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:58.012183 (+ 230us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:37:58.012639 (+ 456us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:37:58.012764 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:58.012860 (+ 96us) server_negotiation.cc:299] Negotiation successful
0619 14:37:58.012907 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 14:37:58.018643 8226 raft_consensus.cc:1261] T 556f712670934445b08946c1a5272461 P 85b530590f0848d99c9df9ee4a75e71c [term 1 FOLLOWER]: Refusing update from remote peer 5a55ef951bdf498b898900f2ebe19989: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:37:58.018643 8368 raft_consensus.cc:1261] T 556f712670934445b08946c1a5272461 P 9f164e7dbb0347039e10be0c23084924 [term 1 FOLLOWER]: Refusing update from remote peer 5a55ef951bdf498b898900f2ebe19989: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:37:58.019328 8453 consensus_queue.cc:1035] T 556f712670934445b08946c1a5272461 P 5a55ef951bdf498b898900f2ebe19989 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f164e7dbb0347039e10be0c23084924" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:37:58.019513 8450 consensus_queue.cc:1035] T 556f712670934445b08946c1a5272461 P 5a55ef951bdf498b898900f2ebe19989 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85b530590f0848d99c9df9ee4a75e71c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:37:58.027325 8457 mvcc.cc:205] Tried to move back new op lower bound from 7298579980357140480 to 7298579980071075840. Current Snapshot: MvccSnapshot[applied={T|T < 7298579980357140480}]
I20260619 14:37:58.029490 8458 mvcc.cc:205] Tried to move back new op lower bound from 7298579980357140480 to 7298579980071075840. Current Snapshot: MvccSnapshot[applied={T|T < 7298579980357140480}]
I20260619 14:37:58.034736 7930 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:37:58.034878 7930 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:37:58.037046 7930 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d7976811491a44f7add55a6e47dc7e97: Sending DeleteTablet for 3 replicas of tablet 556f712670934445b08946c1a5272461
I20260619 14:37:58.037770 8348 tablet_service.cc:1514] Processing DeleteTablet for tablet 556f712670934445b08946c1a5272461 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:37:58 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58214
I20260619 14:37:58.038196 8466 tablet_replica.cc:330] T 556f712670934445b08946c1a5272461 P 9f164e7dbb0347039e10be0c23084924: stopping tablet replica
I20260619 14:37:58.038244 8068 tablet_service.cc:1514] Processing DeleteTablet for tablet 556f712670934445b08946c1a5272461 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:37:58 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60504
I20260619 14:37:58.038528 8466 raft_consensus.cc:2229] T 556f712670934445b08946c1a5272461 P 9f164e7dbb0347039e10be0c23084924 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:37:58.038830 8466 raft_consensus.cc:2258] T 556f712670934445b08946c1a5272461 P 9f164e7dbb0347039e10be0c23084924 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:37:58.038851 8208 tablet_service.cc:1514] Processing DeleteTablet for tablet 556f712670934445b08946c1a5272461 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:37:58 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50132
I20260619 14:37:58.039216 8467 tablet_replica.cc:330] T 556f712670934445b08946c1a5272461 P 5a55ef951bdf498b898900f2ebe19989: stopping tablet replica
I20260619 14:37:58.039475 8467 raft_consensus.cc:2229] T 556f712670934445b08946c1a5272461 P 5a55ef951bdf498b898900f2ebe19989 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:37:58.039417 7930 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56794:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 14:37:58.039810 8467 raft_consensus.cc:2258] T 556f712670934445b08946c1a5272461 P 5a55ef951bdf498b898900f2ebe19989 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 14:37:58.039827 7930 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:37:58.041184 8467 ts_tablet_manager.cc:1905] T 556f712670934445b08946c1a5272461 P 5a55ef951bdf498b898900f2ebe19989: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:37:58.041589 8466 ts_tablet_manager.cc:1905] T 556f712670934445b08946c1a5272461 P 9f164e7dbb0347039e10be0c23084924: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:37:58.044266 8466 ts_tablet_manager.cc:1918] T 556f712670934445b08946c1a5272461 P 9f164e7dbb0347039e10be0c23084924: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:37:58.044359 8466 log.cc:1196] T 556f712670934445b08946c1a5272461 P 9f164e7dbb0347039e10be0c23084924: Deleting WAL directory at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/ts-2/wal/wals/556f712670934445b08946c1a5272461
I20260619 14:37:58.044366 8068 tablet_service.cc:1467] Processing CreateTablet for tablet 4045f361f5cd489788b049d84619e9f8 (DEFAULT_TABLE table=test-table [id=0b4a05ee348241679d2c0691fe53fde6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:37:58.044605 8068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4045f361f5cd489788b049d84619e9f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:58.044663 8466 ts_tablet_manager.cc:1939] T 556f712670934445b08946c1a5272461 P 9f164e7dbb0347039e10be0c23084924: Deleting consensus metadata
I20260619 14:37:58.045907 8468 tablet_replica.cc:330] T 556f712670934445b08946c1a5272461 P 85b530590f0848d99c9df9ee4a75e71c: stopping tablet replica
I20260619 14:37:58.046231 8468 raft_consensus.cc:2229] T 556f712670934445b08946c1a5272461 P 85b530590f0848d99c9df9ee4a75e71c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:37:58.046527 8468 raft_consensus.cc:2258] T 556f712670934445b08946c1a5272461 P 85b530590f0848d99c9df9ee4a75e71c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:37:58.046908 8444 tablet_bootstrap.cc:492] T 4045f361f5cd489788b049d84619e9f8 P 5a55ef951bdf498b898900f2ebe19989: Bootstrap starting.
I20260619 14:37:58.047384 7917 catalog_manager.cc:4841] TS 9f164e7dbb0347039e10be0c23084924 (127.0.98.3:39321): tablet 556f712670934445b08946c1a5272461 (table test-table [id=36488bad5cc54b29892b867120d788b1]) successfully deleted
I20260619 14:37:58.047828 8444 tablet_bootstrap.cc:654] T 4045f361f5cd489788b049d84619e9f8 P 5a55ef951bdf498b898900f2ebe19989: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:58.047904 8208 tablet_service.cc:1467] Processing CreateTablet for tablet 4045f361f5cd489788b049d84619e9f8 (DEFAULT_TABLE table=test-table [id=0b4a05ee348241679d2c0691fe53fde6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:37:58.048028 8348 tablet_service.cc:1467] Processing CreateTablet for tablet 4045f361f5cd489788b049d84619e9f8 (DEFAULT_TABLE table=test-table [id=0b4a05ee348241679d2c0691fe53fde6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:37:58.048143 8208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4045f361f5cd489788b049d84619e9f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:58.048256 8348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4045f361f5cd489788b049d84619e9f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:58.049283 8468 ts_tablet_manager.cc:1905] T 556f712670934445b08946c1a5272461 P 85b530590f0848d99c9df9ee4a75e71c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:37:58.050661 8442 tablet_bootstrap.cc:492] T 4045f361f5cd489788b049d84619e9f8 P 9f164e7dbb0347039e10be0c23084924: Bootstrap starting.
I20260619 14:37:58.050665 8443 tablet_bootstrap.cc:492] T 4045f361f5cd489788b049d84619e9f8 P 85b530590f0848d99c9df9ee4a75e71c: Bootstrap starting.
I20260619 14:37:58.051550 8442 tablet_bootstrap.cc:654] T 4045f361f5cd489788b049d84619e9f8 P 9f164e7dbb0347039e10be0c23084924: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:58.051550 8443 tablet_bootstrap.cc:654] T 4045f361f5cd489788b049d84619e9f8 P 85b530590f0848d99c9df9ee4a75e71c: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:58.051672 8468 ts_tablet_manager.cc:1918] T 556f712670934445b08946c1a5272461 P 85b530590f0848d99c9df9ee4a75e71c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:37:58.051766 8468 log.cc:1196] T 556f712670934445b08946c1a5272461 P 85b530590f0848d99c9df9ee4a75e71c: Deleting WAL directory at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/ts-1/wal/wals/556f712670934445b08946c1a5272461
I20260619 14:37:58.052062 8468 ts_tablet_manager.cc:1939] T 556f712670934445b08946c1a5272461 P 85b530590f0848d99c9df9ee4a75e71c: Deleting consensus metadata
I20260619 14:37:58.053401 7917 catalog_manager.cc:4841] TS 85b530590f0848d99c9df9ee4a75e71c (127.0.98.2:41193): tablet 556f712670934445b08946c1a5272461 (table test-table [id=36488bad5cc54b29892b867120d788b1]) successfully deleted
I20260619 14:37:58.053434 8443 tablet_bootstrap.cc:492] T 4045f361f5cd489788b049d84619e9f8 P 85b530590f0848d99c9df9ee4a75e71c: No bootstrap required, opened a new log
I20260619 14:37:58.053508 8443 ts_tablet_manager.cc:1397] T 4045f361f5cd489788b049d84619e9f8 P 85b530590f0848d99c9df9ee4a75e71c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 14:37:58.053802 8467 ts_tablet_manager.cc:1918] T 556f712670934445b08946c1a5272461 P 5a55ef951bdf498b898900f2ebe19989: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:37:58.053867 8467 log.cc:1196] T 556f712670934445b08946c1a5272461 P 5a55ef951bdf498b898900f2ebe19989: Deleting WAL directory at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/ts-0/wal/wals/556f712670934445b08946c1a5272461
I20260619 14:37:58.053910 8443 raft_consensus.cc:348] T 4045f361f5cd489788b049d84619e9f8 P 85b530590f0848d99c9df9ee4a75e71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a55ef951bdf498b898900f2ebe19989" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } } peers { permanent_uuid: "85b530590f0848d99c9df9ee4a75e71c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41193 } } peers { permanent_uuid: "9f164e7dbb0347039e10be0c23084924" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39321 } }
I20260619 14:37:58.054011 8443 raft_consensus.cc:374] T 4045f361f5cd489788b049d84619e9f8 P 85b530590f0848d99c9df9ee4a75e71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:58.054040 8443 raft_consensus.cc:729] T 4045f361f5cd489788b049d84619e9f8 P 85b530590f0848d99c9df9ee4a75e71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85b530590f0848d99c9df9ee4a75e71c, State: Initialized, Role: FOLLOWER
I20260619 14:37:58.054056 8467 ts_tablet_manager.cc:1939] T 556f712670934445b08946c1a5272461 P 5a55ef951bdf498b898900f2ebe19989: Deleting consensus metadata
I20260619 14:37:58.054153 8443 consensus_queue.cc:260] T 4045f361f5cd489788b049d84619e9f8 P 85b530590f0848d99c9df9ee4a75e71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a55ef951bdf498b898900f2ebe19989" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } } peers { permanent_uuid: "85b530590f0848d99c9df9ee4a75e71c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41193 } } peers { permanent_uuid: "9f164e7dbb0347039e10be0c23084924" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39321 } }
I20260619 14:37:58.054396 8443 ts_tablet_manager.cc:1428] T 4045f361f5cd489788b049d84619e9f8 P 85b530590f0848d99c9df9ee4a75e71c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 14:37:58.055050 7916 catalog_manager.cc:4841] TS 5a55ef951bdf498b898900f2ebe19989 (127.0.98.1:34993): tablet 556f712670934445b08946c1a5272461 (table test-table [id=36488bad5cc54b29892b867120d788b1]) successfully deleted
I20260619 14:37:58.055284 8442 tablet_bootstrap.cc:492] T 4045f361f5cd489788b049d84619e9f8 P 9f164e7dbb0347039e10be0c23084924: No bootstrap required, opened a new log
I20260619 14:37:58.055356 8442 ts_tablet_manager.cc:1397] T 4045f361f5cd489788b049d84619e9f8 P 9f164e7dbb0347039e10be0c23084924: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260619 14:37:58.055749 8442 raft_consensus.cc:348] T 4045f361f5cd489788b049d84619e9f8 P 9f164e7dbb0347039e10be0c23084924 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a55ef951bdf498b898900f2ebe19989" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } } peers { permanent_uuid: "85b530590f0848d99c9df9ee4a75e71c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41193 } } peers { permanent_uuid: "9f164e7dbb0347039e10be0c23084924" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39321 } }
I20260619 14:37:58.055845 8442 raft_consensus.cc:374] T 4045f361f5cd489788b049d84619e9f8 P 9f164e7dbb0347039e10be0c23084924 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:58.055872 8442 raft_consensus.cc:729] T 4045f361f5cd489788b049d84619e9f8 P 9f164e7dbb0347039e10be0c23084924 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f164e7dbb0347039e10be0c23084924, State: Initialized, Role: FOLLOWER
I20260619 14:37:58.055876 8444 tablet_bootstrap.cc:492] T 4045f361f5cd489788b049d84619e9f8 P 5a55ef951bdf498b898900f2ebe19989: No bootstrap required, opened a new log
I20260619 14:37:58.055954 8444 ts_tablet_manager.cc:1397] T 4045f361f5cd489788b049d84619e9f8 P 5a55ef951bdf498b898900f2ebe19989: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20260619 14:37:58.055981 8442 consensus_queue.cc:260] T 4045f361f5cd489788b049d84619e9f8 P 9f164e7dbb0347039e10be0c23084924 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a55ef951bdf498b898900f2ebe19989" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } } peers { permanent_uuid: "85b530590f0848d99c9df9ee4a75e71c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41193 } } peers { permanent_uuid: "9f164e7dbb0347039e10be0c23084924" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39321 } }
I20260619 14:37:58.056200 8442 ts_tablet_manager.cc:1428] T 4045f361f5cd489788b049d84619e9f8 P 9f164e7dbb0347039e10be0c23084924: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:37:58.056365 8444 raft_consensus.cc:348] T 4045f361f5cd489788b049d84619e9f8 P 5a55ef951bdf498b898900f2ebe19989 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a55ef951bdf498b898900f2ebe19989" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } } peers { permanent_uuid: "85b530590f0848d99c9df9ee4a75e71c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41193 } } peers { permanent_uuid: "9f164e7dbb0347039e10be0c23084924" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39321 } }
I20260619 14:37:58.056475 8444 raft_consensus.cc:374] T 4045f361f5cd489788b049d84619e9f8 P 5a55ef951bdf498b898900f2ebe19989 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:58.056509 8444 raft_consensus.cc:729] T 4045f361f5cd489788b049d84619e9f8 P 5a55ef951bdf498b898900f2ebe19989 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a55ef951bdf498b898900f2ebe19989, State: Initialized, Role: FOLLOWER
I20260619 14:37:58.056613 8444 consensus_queue.cc:260] T 4045f361f5cd489788b049d84619e9f8 P 5a55ef951bdf498b898900f2ebe19989 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a55ef951bdf498b898900f2ebe19989" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } } peers { permanent_uuid: "85b530590f0848d99c9df9ee4a75e71c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41193 } } peers { permanent_uuid: "9f164e7dbb0347039e10be0c23084924" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39321 } }
I20260619 14:37:58.056869 8444 ts_tablet_manager.cc:1428] T 4045f361f5cd489788b049d84619e9f8 P 5a55ef951bdf498b898900f2ebe19989: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:37:58.145738 8449 raft_consensus.cc:482] T 4045f361f5cd489788b049d84619e9f8 P 85b530590f0848d99c9df9ee4a75e71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:37:58.145879 8449 raft_consensus.cc:504] T 4045f361f5cd489788b049d84619e9f8 P 85b530590f0848d99c9df9ee4a75e71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a55ef951bdf498b898900f2ebe19989" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } } peers { permanent_uuid: "85b530590f0848d99c9df9ee4a75e71c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41193 } } peers { permanent_uuid: "9f164e7dbb0347039e10be0c23084924" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39321 } }
I20260619 14:37:58.146936 8449 leader_election.cc:288] T 4045f361f5cd489788b049d84619e9f8 P 85b530590f0848d99c9df9ee4a75e71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a55ef951bdf498b898900f2ebe19989 (127.0.98.1:34993), 9f164e7dbb0347039e10be0c23084924 (127.0.98.3:39321)
I20260619 14:37:58.151477 8437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:58.147607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50053 (local address 127.0.98.3:39321)
0619 14:37:58.147720 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:58.147725 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:58.147841 (+ 116us) server_negotiation.cc:407] Connection header received
0619 14:37:58.148008 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:58.148011 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:58.148055 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:58.148096 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:37:58.148445 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:58.149169 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:58.150414 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:58.150868 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:58.150998 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:58.151070 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:58.151268 (+ 198us) server_negotiation.cc:299] Negotiation successful
0619 14:37:58.151348 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 14:37:58.151486 8473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:58.147231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39321 (local address 127.0.98.2:50053)
0619 14:37:58.147767 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:58.147779 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:58.147899 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:58.148212 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:58.148214 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:58.148241 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:37:58.148336 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:58.148341 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:58.149243 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:58.149246 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:58.150260 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:58.150271 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:58.151011 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:58.151015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:58.151123 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:58.151149 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 14:37:58.151245 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 14:37:58.151316 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"mutex_wait_us":65,"thread_start_us":106,"threads_started":1}
I20260619 14:37:58.151983 8472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:58.147172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34993 (local address 127.0.98.2:42563)
0619 14:37:58.147476 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:58.147489 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:58.147646 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:58.148012 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:58.148014 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:58.148028 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:37:58.148156 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:58.148163 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:58.149197 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:58.149201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:58.150609 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:58.150620 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:58.151512 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:58.151516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:58.151616 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:58.151643 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 14:37:58.151766 (+ 123us) client_negotiation.cc:240] Negotiation successful
0619 14:37:58.151835 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":137,"thread_start_us":83,"threads_started":1}
I20260619 14:37:58.152012 8440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:58.147267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42563 (local address 127.0.98.1:34993)
0619 14:37:58.147404 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:58.147408 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:58.147563 (+ 155us) server_negotiation.cc:407] Connection header received
0619 14:37:58.147779 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:58.147783 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:58.147835 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:58.147882 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:37:58.148271 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:58.149071 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:58.150754 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:58.151269 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:58.151373 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:58.151437 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:58.151809 (+ 372us) server_negotiation.cc:299] Negotiation successful
0619 14:37:58.151896 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 14:37:58.152254 8368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4045f361f5cd489788b049d84619e9f8" candidate_uuid: "85b530590f0848d99c9df9ee4a75e71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f164e7dbb0347039e10be0c23084924" is_pre_election: true
I20260619 14:37:58.152390 8368 raft_consensus.cc:2454] T 4045f361f5cd489788b049d84619e9f8 P 9f164e7dbb0347039e10be0c23084924 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85b530590f0848d99c9df9ee4a75e71c in term 0.
I20260619 14:37:58.152786 8161 leader_election.cc:302] T 4045f361f5cd489788b049d84619e9f8 P 85b530590f0848d99c9df9ee4a75e71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85b530590f0848d99c9df9ee4a75e71c, 9f164e7dbb0347039e10be0c23084924; no voters:
I20260619 14:37:58.152936 8088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4045f361f5cd489788b049d84619e9f8" candidate_uuid: "85b530590f0848d99c9df9ee4a75e71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a55ef951bdf498b898900f2ebe19989" is_pre_election: true
I20260619 14:37:58.153102 8449 raft_consensus.cc:2781] T 4045f361f5cd489788b049d84619e9f8 P 85b530590f0848d99c9df9ee4a75e71c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:37:58.153159 8449 raft_consensus.cc:482] T 4045f361f5cd489788b049d84619e9f8 P 85b530590f0848d99c9df9ee4a75e71c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:37:58.153199 8449 raft_consensus.cc:3037] T 4045f361f5cd489788b049d84619e9f8 P 85b530590f0848d99c9df9ee4a75e71c [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:58.153172 8088 raft_consensus.cc:2454] T 4045f361f5cd489788b049d84619e9f8 P 5a55ef951bdf498b898900f2ebe19989 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85b530590f0848d99c9df9ee4a75e71c in term 0.
I20260619 14:37:58.154305 8449 raft_consensus.cc:504] T 4045f361f5cd489788b049d84619e9f8 P 85b530590f0848d99c9df9ee4a75e71c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a55ef951bdf498b898900f2ebe19989" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } } peers { permanent_uuid: "85b530590f0848d99c9df9ee4a75e71c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41193 } } peers { permanent_uuid: "9f164e7dbb0347039e10be0c23084924" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39321 } }
I20260619 14:37:58.154645 8449 leader_election.cc:288] T 4045f361f5cd489788b049d84619e9f8 P 85b530590f0848d99c9df9ee4a75e71c [CANDIDATE]: Term 1 election: Requested vote from peers 5a55ef951bdf498b898900f2ebe19989 (127.0.98.1:34993), 9f164e7dbb0347039e10be0c23084924 (127.0.98.3:39321)
I20260619 14:37:58.155040 8088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4045f361f5cd489788b049d84619e9f8" candidate_uuid: "85b530590f0848d99c9df9ee4a75e71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a55ef951bdf498b898900f2ebe19989"
I20260619 14:37:58.155055 8368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4045f361f5cd489788b049d84619e9f8" candidate_uuid: "85b530590f0848d99c9df9ee4a75e71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f164e7dbb0347039e10be0c23084924"
I20260619 14:37:58.155169 8368 raft_consensus.cc:3037] T 4045f361f5cd489788b049d84619e9f8 P 9f164e7dbb0347039e10be0c23084924 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:58.155169 8088 raft_consensus.cc:3037] T 4045f361f5cd489788b049d84619e9f8 P 5a55ef951bdf498b898900f2ebe19989 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:58.156297 8368 raft_consensus.cc:2454] T 4045f361f5cd489788b049d84619e9f8 P 9f164e7dbb0347039e10be0c23084924 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85b530590f0848d99c9df9ee4a75e71c in term 1.
I20260619 14:37:58.156297 8088 raft_consensus.cc:2454] T 4045f361f5cd489788b049d84619e9f8 P 5a55ef951bdf498b898900f2ebe19989 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85b530590f0848d99c9df9ee4a75e71c in term 1.
I20260619 14:37:58.156625 8160 leader_election.cc:302] T 4045f361f5cd489788b049d84619e9f8 P 85b530590f0848d99c9df9ee4a75e71c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a55ef951bdf498b898900f2ebe19989, 85b530590f0848d99c9df9ee4a75e71c; no voters:
I20260619 14:37:58.156874 8449 raft_consensus.cc:2781] T 4045f361f5cd489788b049d84619e9f8 P 85b530590f0848d99c9df9ee4a75e71c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:37:58.157109 8449 raft_consensus.cc:686] T 4045f361f5cd489788b049d84619e9f8 P 85b530590f0848d99c9df9ee4a75e71c [term 1 LEADER]: Becoming Leader. State: Replica: 85b530590f0848d99c9df9ee4a75e71c, State: Running, Role: LEADER
I20260619 14:37:58.157361 8449 consensus_queue.cc:237] T 4045f361f5cd489788b049d84619e9f8 P 85b530590f0848d99c9df9ee4a75e71c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a55ef951bdf498b898900f2ebe19989" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } } peers { permanent_uuid: "85b530590f0848d99c9df9ee4a75e71c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41193 } } peers { permanent_uuid: "9f164e7dbb0347039e10be0c23084924" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39321 } }
I20260619 14:37:58.159758 7931 catalog_manager.cc:5495] T 4045f361f5cd489788b049d84619e9f8 P 85b530590f0848d99c9df9ee4a75e71c reported cstate change: term changed from 0 to 1, leader changed from <none> to 85b530590f0848d99c9df9ee4a75e71c (127.0.98.2). New cstate: current_term: 1 leader_uuid: "85b530590f0848d99c9df9ee4a75e71c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a55ef951bdf498b898900f2ebe19989" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85b530590f0848d99c9df9ee4a75e71c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f164e7dbb0347039e10be0c23084924" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39321 } health_report { overall_health: UNKNOWN } } }
Jun 19 14:37:58 dist-test-slave-32hj krb5kdc[7886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879876, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 14:37:58.197553 8481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:58.190089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44517 (local address 127.0.0.1:50320)
0619 14:37:58.190342 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:58.190354 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:58.190443 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:58.190648 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:58.190650 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:58.190995 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:37:58.191090 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:58.191096 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:58.191802 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:58.191804 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:58.192358 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:58.192365 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:58.192777 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:58.192781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:58.192858 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:58.193397 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:37:58.193408 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:37:58.194750 (+ 1342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:37:58.196126 (+ 1376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:58.196134 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:58.196140 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:58.196351 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:58.196573 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:37:58.196576 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:37:58.196579 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:37:58.196671 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:37:58.196999 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:37:58.197006 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:37:58.197108 (+ 102us) client_negotiation.cc:769] Sending connection context
0619 14:37:58.197218 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 14:37:58.197369 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":107,"threads_started":1}
I20260619 14:37:58.197553 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:58.190161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50320 (local address 127.0.98.62:44517)
0619 14:37:58.190282 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:58.190286 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:58.190415 (+ 129us) server_negotiation.cc:407] Connection header received
0619 14:37:58.190520 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:58.190522 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:58.190560 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:58.190615 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:58.191186 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:58.191727 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:58.192440 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:58.192647 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:58.192775 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:58.194864 (+ 2089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:58.194868 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:58.194870 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:58.194907 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:58.196037 (+ 1130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:58.196433 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:58.196436 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:58.196438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:58.196500 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:58.196758 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:58.196761 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:58.196763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:58.196911 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:58.196990 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:58.197274 (+ 284us) server_negotiation.cc:299] Negotiation successful
0619 14:37:58.197418 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":51}
I20260619 14:37:58.200776 7931 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50320:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 14:37:58.201421 7931 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:37:58.205711 8068 tablet_service.cc:1467] Processing CreateTablet for tablet 35e26d16e5c24045be397dd9d5529ed9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=13d813a19ea44720b8673a843da11da6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 14:37:58.205840 8208 tablet_service.cc:1467] Processing CreateTablet for tablet 35e26d16e5c24045be397dd9d5529ed9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=13d813a19ea44720b8673a843da11da6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 14:37:58.205873 8348 tablet_service.cc:1467] Processing CreateTablet for tablet 35e26d16e5c24045be397dd9d5529ed9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=13d813a19ea44720b8673a843da11da6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 14:37:58.206102 8068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35e26d16e5c24045be397dd9d5529ed9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:58.206172 8348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35e26d16e5c24045be397dd9d5529ed9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:58.206183 8208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35e26d16e5c24045be397dd9d5529ed9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:58.208345 8444 tablet_bootstrap.cc:492] T 35e26d16e5c24045be397dd9d5529ed9 P 5a55ef951bdf498b898900f2ebe19989: Bootstrap starting.
I20260619 14:37:58.208801 8442 tablet_bootstrap.cc:492] T 35e26d16e5c24045be397dd9d5529ed9 P 9f164e7dbb0347039e10be0c23084924: Bootstrap starting.
I20260619 14:37:58.208833 8443 tablet_bootstrap.cc:492] T 35e26d16e5c24045be397dd9d5529ed9 P 85b530590f0848d99c9df9ee4a75e71c: Bootstrap starting.
I20260619 14:37:58.209173 8444 tablet_bootstrap.cc:654] T 35e26d16e5c24045be397dd9d5529ed9 P 5a55ef951bdf498b898900f2ebe19989: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:58.209554 8442 tablet_bootstrap.cc:654] T 35e26d16e5c24045be397dd9d5529ed9 P 9f164e7dbb0347039e10be0c23084924: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:58.209726 8443 tablet_bootstrap.cc:654] T 35e26d16e5c24045be397dd9d5529ed9 P 85b530590f0848d99c9df9ee4a75e71c: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:58.210338 8444 tablet_bootstrap.cc:492] T 35e26d16e5c24045be397dd9d5529ed9 P 5a55ef951bdf498b898900f2ebe19989: No bootstrap required, opened a new log
I20260619 14:37:58.210417 8444 ts_tablet_manager.cc:1397] T 35e26d16e5c24045be397dd9d5529ed9 P 5a55ef951bdf498b898900f2ebe19989: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 14:37:58.210608 8442 tablet_bootstrap.cc:492] T 35e26d16e5c24045be397dd9d5529ed9 P 9f164e7dbb0347039e10be0c23084924: No bootstrap required, opened a new log
I20260619 14:37:58.210697 8442 ts_tablet_manager.cc:1397] T 35e26d16e5c24045be397dd9d5529ed9 P 9f164e7dbb0347039e10be0c23084924: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 14:37:58.210814 8444 raft_consensus.cc:348] T 35e26d16e5c24045be397dd9d5529ed9 P 5a55ef951bdf498b898900f2ebe19989 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a55ef951bdf498b898900f2ebe19989" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } } peers { permanent_uuid: "85b530590f0848d99c9df9ee4a75e71c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41193 } } peers { permanent_uuid: "9f164e7dbb0347039e10be0c23084924" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39321 } }
I20260619 14:37:58.210893 8443 tablet_bootstrap.cc:492] T 35e26d16e5c24045be397dd9d5529ed9 P 85b530590f0848d99c9df9ee4a75e71c: No bootstrap required, opened a new log
I20260619 14:37:58.210924 8444 raft_consensus.cc:374] T 35e26d16e5c24045be397dd9d5529ed9 P 5a55ef951bdf498b898900f2ebe19989 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:58.210952 8444 raft_consensus.cc:729] T 35e26d16e5c24045be397dd9d5529ed9 P 5a55ef951bdf498b898900f2ebe19989 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a55ef951bdf498b898900f2ebe19989, State: Initialized, Role: FOLLOWER
I20260619 14:37:58.210959 8443 ts_tablet_manager.cc:1397] T 35e26d16e5c24045be397dd9d5529ed9 P 85b530590f0848d99c9df9ee4a75e71c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 14:37:58.211087 8444 consensus_queue.cc:260] T 35e26d16e5c24045be397dd9d5529ed9 P 5a55ef951bdf498b898900f2ebe19989 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a55ef951bdf498b898900f2ebe19989" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } } peers { permanent_uuid: "85b530590f0848d99c9df9ee4a75e71c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41193 } } peers { permanent_uuid: "9f164e7dbb0347039e10be0c23084924" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39321 } }
I20260619 14:37:58.211100 8442 raft_consensus.cc:348] T 35e26d16e5c24045be397dd9d5529ed9 P 9f164e7dbb0347039e10be0c23084924 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a55ef951bdf498b898900f2ebe19989" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } } peers { permanent_uuid: "85b530590f0848d99c9df9ee4a75e71c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41193 } } peers { permanent_uuid: "9f164e7dbb0347039e10be0c23084924" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39321 } }
I20260619 14:37:58.211202 8442 raft_consensus.cc:374] T 35e26d16e5c24045be397dd9d5529ed9 P 9f164e7dbb0347039e10be0c23084924 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:58.211236 8442 raft_consensus.cc:729] T 35e26d16e5c24045be397dd9d5529ed9 P 9f164e7dbb0347039e10be0c23084924 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f164e7dbb0347039e10be0c23084924, State: Initialized, Role: FOLLOWER
I20260619 14:37:58.211340 8442 consensus_queue.cc:260] T 35e26d16e5c24045be397dd9d5529ed9 P 9f164e7dbb0347039e10be0c23084924 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a55ef951bdf498b898900f2ebe19989" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } } peers { permanent_uuid: "85b530590f0848d99c9df9ee4a75e71c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41193 } } peers { permanent_uuid: "9f164e7dbb0347039e10be0c23084924" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39321 } }
I20260619 14:37:58.211330 8450 tablet_replica.cc:439] T 35e26d16e5c24045be397dd9d5529ed9 P 5a55ef951bdf498b898900f2ebe19989: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a55ef951bdf498b898900f2ebe19989" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } } peers { permanent_uuid: "85b530590f0848d99c9df9ee4a75e71c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41193 } } peers { permanent_uuid: "9f164e7dbb0347039e10be0c23084924" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39321 } } }
I20260619 14:37:58.211349 8443 raft_consensus.cc:348] T 35e26d16e5c24045be397dd9d5529ed9 P 85b530590f0848d99c9df9ee4a75e71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a55ef951bdf498b898900f2ebe19989" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } } peers { permanent_uuid: "85b530590f0848d99c9df9ee4a75e71c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41193 } } peers { permanent_uuid: "9f164e7dbb0347039e10be0c23084924" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39321 } }
I20260619 14:37:58.211440 8443 raft_consensus.cc:374] T 35e26d16e5c24045be397dd9d5529ed9 P 85b530590f0848d99c9df9ee4a75e71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:58.211455 8450 tablet_replica.cc:442] T 35e26d16e5c24045be397dd9d5529ed9 P 5a55ef951bdf498b898900f2ebe19989: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:37:58.211467 8443 raft_consensus.cc:729] T 35e26d16e5c24045be397dd9d5529ed9 P 85b530590f0848d99c9df9ee4a75e71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85b530590f0848d99c9df9ee4a75e71c, State: Initialized, Role: FOLLOWER
I20260619 14:37:58.211575 8442 ts_tablet_manager.cc:1428] T 35e26d16e5c24045be397dd9d5529ed9 P 9f164e7dbb0347039e10be0c23084924: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:37:58.211570 8443 consensus_queue.cc:260] T 35e26d16e5c24045be397dd9d5529ed9 P 85b530590f0848d99c9df9ee4a75e71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a55ef951bdf498b898900f2ebe19989" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } } peers { permanent_uuid: "85b530590f0848d99c9df9ee4a75e71c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41193 } } peers { permanent_uuid: "9f164e7dbb0347039e10be0c23084924" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39321 } }
I20260619 14:37:58.211691 8448 tablet_replica.cc:439] T 35e26d16e5c24045be397dd9d5529ed9 P 9f164e7dbb0347039e10be0c23084924: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a55ef951bdf498b898900f2ebe19989" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } } peers { permanent_uuid: "85b530590f0848d99c9df9ee4a75e71c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41193 } } peers { permanent_uuid: "9f164e7dbb0347039e10be0c23084924" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39321 } } }
I20260619 14:37:58.211802 8443 ts_tablet_manager.cc:1428] T 35e26d16e5c24045be397dd9d5529ed9 P 85b530590f0848d99c9df9ee4a75e71c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:37:58.211825 8448 tablet_replica.cc:442] T 35e26d16e5c24045be397dd9d5529ed9 P 9f164e7dbb0347039e10be0c23084924: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:37:58.211334 8444 ts_tablet_manager.cc:1428] T 35e26d16e5c24045be397dd9d5529ed9 P 5a55ef951bdf498b898900f2ebe19989: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:37:58.212011 8474 tablet_replica.cc:439] T 35e26d16e5c24045be397dd9d5529ed9 P 85b530590f0848d99c9df9ee4a75e71c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a55ef951bdf498b898900f2ebe19989" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } } peers { permanent_uuid: "85b530590f0848d99c9df9ee4a75e71c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41193 } } peers { permanent_uuid: "9f164e7dbb0347039e10be0c23084924" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39321 } } }
I20260619 14:37:58.212250 8474 tablet_replica.cc:442] T 35e26d16e5c24045be397dd9d5529ed9 P 85b530590f0848d99c9df9ee4a75e71c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:37:58.330504 8474 raft_consensus.cc:482] T 35e26d16e5c24045be397dd9d5529ed9 P 85b530590f0848d99c9df9ee4a75e71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:37:58.330646 8474 raft_consensus.cc:504] T 35e26d16e5c24045be397dd9d5529ed9 P 85b530590f0848d99c9df9ee4a75e71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a55ef951bdf498b898900f2ebe19989" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } } peers { permanent_uuid: "85b530590f0848d99c9df9ee4a75e71c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41193 } } peers { permanent_uuid: "9f164e7dbb0347039e10be0c23084924" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39321 } }
I20260619 14:37:58.331064 8474 leader_election.cc:288] T 35e26d16e5c24045be397dd9d5529ed9 P 85b530590f0848d99c9df9ee4a75e71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a55ef951bdf498b898900f2ebe19989 (127.0.98.1:34993), 9f164e7dbb0347039e10be0c23084924 (127.0.98.3:39321)
I20260619 14:37:58.331499 8368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35e26d16e5c24045be397dd9d5529ed9" candidate_uuid: "85b530590f0848d99c9df9ee4a75e71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f164e7dbb0347039e10be0c23084924" is_pre_election: true
I20260619 14:37:58.331499 8088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35e26d16e5c24045be397dd9d5529ed9" candidate_uuid: "85b530590f0848d99c9df9ee4a75e71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a55ef951bdf498b898900f2ebe19989" is_pre_election: true
I20260619 14:37:58.331630 8368 raft_consensus.cc:2454] T 35e26d16e5c24045be397dd9d5529ed9 P 9f164e7dbb0347039e10be0c23084924 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85b530590f0848d99c9df9ee4a75e71c in term 0.
I20260619 14:37:58.331673 8088 raft_consensus.cc:2454] T 35e26d16e5c24045be397dd9d5529ed9 P 5a55ef951bdf498b898900f2ebe19989 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85b530590f0848d99c9df9ee4a75e71c in term 0.
I20260619 14:37:58.331907 8161 leader_election.cc:302] T 35e26d16e5c24045be397dd9d5529ed9 P 85b530590f0848d99c9df9ee4a75e71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85b530590f0848d99c9df9ee4a75e71c, 9f164e7dbb0347039e10be0c23084924; no voters:
I20260619 14:37:58.332077 8474 raft_consensus.cc:2781] T 35e26d16e5c24045be397dd9d5529ed9 P 85b530590f0848d99c9df9ee4a75e71c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:37:58.332140 8474 raft_consensus.cc:482] T 35e26d16e5c24045be397dd9d5529ed9 P 85b530590f0848d99c9df9ee4a75e71c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:37:58.332167 8474 raft_consensus.cc:3037] T 35e26d16e5c24045be397dd9d5529ed9 P 85b530590f0848d99c9df9ee4a75e71c [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:58.333124 8474 raft_consensus.cc:504] T 35e26d16e5c24045be397dd9d5529ed9 P 85b530590f0848d99c9df9ee4a75e71c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a55ef951bdf498b898900f2ebe19989" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } } peers { permanent_uuid: "85b530590f0848d99c9df9ee4a75e71c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41193 } } peers { permanent_uuid: "9f164e7dbb0347039e10be0c23084924" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39321 } }
I20260619 14:37:58.333415 8474 leader_election.cc:288] T 35e26d16e5c24045be397dd9d5529ed9 P 85b530590f0848d99c9df9ee4a75e71c [CANDIDATE]: Term 1 election: Requested vote from peers 5a55ef951bdf498b898900f2ebe19989 (127.0.98.1:34993), 9f164e7dbb0347039e10be0c23084924 (127.0.98.3:39321)
I20260619 14:37:58.333734 8088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35e26d16e5c24045be397dd9d5529ed9" candidate_uuid: "85b530590f0848d99c9df9ee4a75e71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a55ef951bdf498b898900f2ebe19989"
I20260619 14:37:58.333848 8088 raft_consensus.cc:3037] T 35e26d16e5c24045be397dd9d5529ed9 P 5a55ef951bdf498b898900f2ebe19989 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:58.333827 8368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35e26d16e5c24045be397dd9d5529ed9" candidate_uuid: "85b530590f0848d99c9df9ee4a75e71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f164e7dbb0347039e10be0c23084924"
I20260619 14:37:58.333942 8368 raft_consensus.cc:3037] T 35e26d16e5c24045be397dd9d5529ed9 P 9f164e7dbb0347039e10be0c23084924 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:58.334735 8368 raft_consensus.cc:2454] T 35e26d16e5c24045be397dd9d5529ed9 P 9f164e7dbb0347039e10be0c23084924 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85b530590f0848d99c9df9ee4a75e71c in term 1.
I20260619 14:37:58.334738 8088 raft_consensus.cc:2454] T 35e26d16e5c24045be397dd9d5529ed9 P 5a55ef951bdf498b898900f2ebe19989 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85b530590f0848d99c9df9ee4a75e71c in term 1.
I20260619 14:37:58.334969 8160 leader_election.cc:302] T 35e26d16e5c24045be397dd9d5529ed9 P 85b530590f0848d99c9df9ee4a75e71c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a55ef951bdf498b898900f2ebe19989, 85b530590f0848d99c9df9ee4a75e71c; no voters:
I20260619 14:37:58.335120 8474 raft_consensus.cc:2781] T 35e26d16e5c24045be397dd9d5529ed9 P 85b530590f0848d99c9df9ee4a75e71c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:37:58.335206 8474 raft_consensus.cc:686] T 35e26d16e5c24045be397dd9d5529ed9 P 85b530590f0848d99c9df9ee4a75e71c [term 1 LEADER]: Becoming Leader. State: Replica: 85b530590f0848d99c9df9ee4a75e71c, State: Running, Role: LEADER
I20260619 14:37:58.335359 8474 consensus_queue.cc:237] T 35e26d16e5c24045be397dd9d5529ed9 P 85b530590f0848d99c9df9ee4a75e71c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a55ef951bdf498b898900f2ebe19989" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } } peers { permanent_uuid: "85b530590f0848d99c9df9ee4a75e71c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41193 } } peers { permanent_uuid: "9f164e7dbb0347039e10be0c23084924" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39321 } }
I20260619 14:37:58.335948 8449 tablet_replica.cc:439] T 35e26d16e5c24045be397dd9d5529ed9 P 85b530590f0848d99c9df9ee4a75e71c: TxnStatusTablet state changed. Reason: New leader 85b530590f0848d99c9df9ee4a75e71c. Latest consensus state: current_term: 1 leader_uuid: "85b530590f0848d99c9df9ee4a75e71c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a55ef951bdf498b898900f2ebe19989" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } } peers { permanent_uuid: "85b530590f0848d99c9df9ee4a75e71c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41193 } } peers { permanent_uuid: "9f164e7dbb0347039e10be0c23084924" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39321 } } }
I20260619 14:37:58.336052 8449 tablet_replica.cc:442] T 35e26d16e5c24045be397dd9d5529ed9 P 85b530590f0848d99c9df9ee4a75e71c: This TxnStatusTablet replica's current role is: LEADER
I20260619 14:37:58.336330 8486 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 14:37:58.336485 8486 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 14:37:58.336994 7930 catalog_manager.cc:5495] T 35e26d16e5c24045be397dd9d5529ed9 P 85b530590f0848d99c9df9ee4a75e71c reported cstate change: term changed from 0 to 1, leader changed from <none> to 85b530590f0848d99c9df9ee4a75e71c (127.0.98.2). New cstate: current_term: 1 leader_uuid: "85b530590f0848d99c9df9ee4a75e71c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a55ef951bdf498b898900f2ebe19989" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85b530590f0848d99c9df9ee4a75e71c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f164e7dbb0347039e10be0c23084924" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39321 } health_report { overall_health: UNKNOWN } } }
I20260619 14:37:58.364069 8481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:58.360607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41193 (local address 127.0.0.1:44234)
0619 14:37:58.360765 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:58.360781 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:58.360884 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:58.361179 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:58.361182 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:58.361190 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:37:58.361291 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:58.361296 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:58.361962 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:58.361965 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:58.362598 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:58.362606 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:58.363003 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:58.363007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:58.363094 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:58.363134 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:37:58.363775 (+ 641us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:37:58.363803 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 14:37:58.363860 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 14:37:58.363943 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260619 14:37:58.364089 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:58.360757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44234 (local address 127.0.98.2:41193)
0619 14:37:58.360871 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:58.360876 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:58.360891 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:37:58.361012 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:58.361016 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:58.361075 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:58.361154 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:37:58.361385 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:58.361868 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:58.362700 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:58.362906 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:58.362989 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:58.363291 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:37:58.363688 (+ 397us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:37:58.363788 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:58.363879 (+ 91us) server_negotiation.cc:299] Negotiation successful
0619 14:37:58.363930 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 14:37:58.366829 8368 raft_consensus.cc:1261] T 35e26d16e5c24045be397dd9d5529ed9 P 9f164e7dbb0347039e10be0c23084924 [term 1 FOLLOWER]: Refusing update from remote peer 85b530590f0848d99c9df9ee4a75e71c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:37:58.367059 8088 raft_consensus.cc:1261] T 35e26d16e5c24045be397dd9d5529ed9 P 5a55ef951bdf498b898900f2ebe19989 [term 1 FOLLOWER]: Refusing update from remote peer 85b530590f0848d99c9df9ee4a75e71c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:37:58.367292 8474 consensus_queue.cc:1035] T 35e26d16e5c24045be397dd9d5529ed9 P 85b530590f0848d99c9df9ee4a75e71c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f164e7dbb0347039e10be0c23084924" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:37:58.367518 8449 consensus_queue.cc:1035] T 35e26d16e5c24045be397dd9d5529ed9 P 85b530590f0848d99c9df9ee4a75e71c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a55ef951bdf498b898900f2ebe19989" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:37:58.368957 8450 tablet_replica.cc:439] T 35e26d16e5c24045be397dd9d5529ed9 P 5a55ef951bdf498b898900f2ebe19989: TxnStatusTablet state changed. Reason: New leader 85b530590f0848d99c9df9ee4a75e71c. Latest consensus state: current_term: 1 leader_uuid: "85b530590f0848d99c9df9ee4a75e71c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a55ef951bdf498b898900f2ebe19989" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } } peers { permanent_uuid: "85b530590f0848d99c9df9ee4a75e71c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41193 } } peers { permanent_uuid: "9f164e7dbb0347039e10be0c23084924" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39321 } } }
I20260619 14:37:58.369071 8450 tablet_replica.cc:442] T 35e26d16e5c24045be397dd9d5529ed9 P 5a55ef951bdf498b898900f2ebe19989: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:37:58.369338 8448 tablet_replica.cc:439] T 35e26d16e5c24045be397dd9d5529ed9 P 9f164e7dbb0347039e10be0c23084924: TxnStatusTablet state changed. Reason: New leader 85b530590f0848d99c9df9ee4a75e71c. Latest consensus state: current_term: 1 leader_uuid: "85b530590f0848d99c9df9ee4a75e71c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a55ef951bdf498b898900f2ebe19989" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } } peers { permanent_uuid: "85b530590f0848d99c9df9ee4a75e71c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41193 } } peers { permanent_uuid: "9f164e7dbb0347039e10be0c23084924" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39321 } } }
I20260619 14:37:58.369445 8448 tablet_replica.cc:442] T 35e26d16e5c24045be397dd9d5529ed9 P 9f164e7dbb0347039e10be0c23084924: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:37:58.371214 8449 tablet_replica.cc:439] T 35e26d16e5c24045be397dd9d5529ed9 P 85b530590f0848d99c9df9ee4a75e71c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "85b530590f0848d99c9df9ee4a75e71c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a55ef951bdf498b898900f2ebe19989" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } } peers { permanent_uuid: "85b530590f0848d99c9df9ee4a75e71c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41193 } } peers { permanent_uuid: "9f164e7dbb0347039e10be0c23084924" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39321 } } }
I20260619 14:37:58.371313 8449 tablet_replica.cc:442] T 35e26d16e5c24045be397dd9d5529ed9 P 85b530590f0848d99c9df9ee4a75e71c: This TxnStatusTablet replica's current role is: LEADER
I20260619 14:37:58.372119 8448 tablet_replica.cc:439] T 35e26d16e5c24045be397dd9d5529ed9 P 9f164e7dbb0347039e10be0c23084924: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "85b530590f0848d99c9df9ee4a75e71c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a55ef951bdf498b898900f2ebe19989" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } } peers { permanent_uuid: "85b530590f0848d99c9df9ee4a75e71c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41193 } } peers { permanent_uuid: "9f164e7dbb0347039e10be0c23084924" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39321 } } }
I20260619 14:37:58.372238 8448 tablet_replica.cc:442] T 35e26d16e5c24045be397dd9d5529ed9 P 9f164e7dbb0347039e10be0c23084924: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:37:58.374984 8474 tablet_replica.cc:439] T 35e26d16e5c24045be397dd9d5529ed9 P 85b530590f0848d99c9df9ee4a75e71c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "85b530590f0848d99c9df9ee4a75e71c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a55ef951bdf498b898900f2ebe19989" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } } peers { permanent_uuid: "85b530590f0848d99c9df9ee4a75e71c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41193 } } peers { permanent_uuid: "9f164e7dbb0347039e10be0c23084924" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39321 } } }
I20260619 14:37:58.375092 8474 tablet_replica.cc:442] T 35e26d16e5c24045be397dd9d5529ed9 P 85b530590f0848d99c9df9ee4a75e71c: This TxnStatusTablet replica's current role is: LEADER
I20260619 14:37:58.376086 8456 mvcc.cc:205] Tried to move back new op lower bound from 7298579981785800704 to 7298579981662720000. Current Snapshot: MvccSnapshot[applied={T|T < 7298579981785800704 or (T in {7298579981785800704})}]
I20260619 14:37:58.376746 8450 tablet_replica.cc:439] T 35e26d16e5c24045be397dd9d5529ed9 P 5a55ef951bdf498b898900f2ebe19989: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "85b530590f0848d99c9df9ee4a75e71c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a55ef951bdf498b898900f2ebe19989" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } } peers { permanent_uuid: "85b530590f0848d99c9df9ee4a75e71c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41193 } } peers { permanent_uuid: "9f164e7dbb0347039e10be0c23084924" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39321 } } }
I20260619 14:37:58.376858 8450 tablet_replica.cc:442] T 35e26d16e5c24045be397dd9d5529ed9 P 5a55ef951bdf498b898900f2ebe19989: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:37:58.379416 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:58.376283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44236 (local address 127.0.98.2:41193)
0619 14:37:58.376378 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:58.376382 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:58.376394 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:37:58.376433 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:58.376435 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:58.376478 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:58.376546 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:37:58.376832 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:58.377513 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:58.378458 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:58.378742 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:58.378829 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:58.379050 (+ 221us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:37:58.379118 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:37:58.379172 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:58.379257 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 14:37:58.379301 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 14:37:58.388775 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:58.385832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59689 (local address 127.0.98.2:41193)
0619 14:37:58.385921 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:58.385925 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:58.386126 (+ 201us) server_negotiation.cc:407] Connection header received
0619 14:37:58.386221 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:58.386224 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:58.386259 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:58.386313 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:37:58.386509 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:58.386967 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:58.387859 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:58.388053 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:58.388129 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:58.388373 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:37:58.388429 (+ 56us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:37:58.388483 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:58.388561 (+ 78us) server_negotiation.cc:299] Negotiation successful
0619 14:37:58.388612 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":32}
I20260619 14:37:58.388793 8494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:58.385796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41193 (local address 127.0.98.2:59689)
0619 14:37:58.386051 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:58.386063 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:58.386153 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:58.386324 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:58.386326 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:58.386333 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:37:58.386426 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:58.386432 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:58.387060 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:58.387063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:58.387762 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:58.387772 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:58.388152 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:58.388156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:58.388235 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:58.388269 (+ 34us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:37:58.388492 (+ 223us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:37:58.388498 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 14:37:58.388552 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 14:37:58.388644 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":16,"thread_start_us":102,"threads_started":1}
I20260619 14:37:58.394754 8368 raft_consensus.cc:1261] T 4045f361f5cd489788b049d84619e9f8 P 9f164e7dbb0347039e10be0c23084924 [term 1 FOLLOWER]: Refusing update from remote peer 85b530590f0848d99c9df9ee4a75e71c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:37:58.395051 8088 raft_consensus.cc:1261] T 4045f361f5cd489788b049d84619e9f8 P 5a55ef951bdf498b898900f2ebe19989 [term 1 FOLLOWER]: Refusing update from remote peer 85b530590f0848d99c9df9ee4a75e71c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:37:58.395143 8474 consensus_queue.cc:1035] T 4045f361f5cd489788b049d84619e9f8 P 85b530590f0848d99c9df9ee4a75e71c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f164e7dbb0347039e10be0c23084924" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:37:58.395387 8474 consensus_queue.cc:1035] T 4045f361f5cd489788b049d84619e9f8 P 85b530590f0848d99c9df9ee4a75e71c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a55ef951bdf498b898900f2ebe19989" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:37:58.435643 7931 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:37:58.435822 7931 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:37:58.437839 7931 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d7976811491a44f7add55a6e47dc7e97: Sending DeleteTablet for 3 replicas of tablet 4045f361f5cd489788b049d84619e9f8
I20260619 14:37:58.438366 8068 tablet_service.cc:1514] Processing DeleteTablet for tablet 4045f361f5cd489788b049d84619e9f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:37:58 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60504
I20260619 14:37:58.438431 8208 tablet_service.cc:1514] Processing DeleteTablet for tablet 4045f361f5cd489788b049d84619e9f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:37:58 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50132
I20260619 14:37:58.438512 8348 tablet_service.cc:1514] Processing DeleteTablet for tablet 4045f361f5cd489788b049d84619e9f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:37:58 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58214
I20260619 14:37:58.438609 8468 tablet_replica.cc:330] T 4045f361f5cd489788b049d84619e9f8 P 85b530590f0848d99c9df9ee4a75e71c: stopping tablet replica
I20260619 14:37:58.438680 8467 tablet_replica.cc:330] T 4045f361f5cd489788b049d84619e9f8 P 5a55ef951bdf498b898900f2ebe19989: stopping tablet replica
I20260619 14:37:58.438735 8468 raft_consensus.cc:2229] T 4045f361f5cd489788b049d84619e9f8 P 85b530590f0848d99c9df9ee4a75e71c [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:37:58.438791 8467 raft_consensus.cc:2229] T 4045f361f5cd489788b049d84619e9f8 P 5a55ef951bdf498b898900f2ebe19989 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:37:58.438902 8467 raft_consensus.cc:2258] T 4045f361f5cd489788b049d84619e9f8 P 5a55ef951bdf498b898900f2ebe19989 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:37:58.439023 8468 raft_consensus.cc:2258] T 4045f361f5cd489788b049d84619e9f8 P 85b530590f0848d99c9df9ee4a75e71c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:37:58.439059 8466 tablet_replica.cc:330] T 4045f361f5cd489788b049d84619e9f8 P 9f164e7dbb0347039e10be0c23084924: stopping tablet replica
I20260619 14:37:58.439153 8466 raft_consensus.cc:2229] T 4045f361f5cd489788b049d84619e9f8 P 9f164e7dbb0347039e10be0c23084924 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:37:58.439280 8466 raft_consensus.cc:2258] T 4045f361f5cd489788b049d84619e9f8 P 9f164e7dbb0347039e10be0c23084924 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:37:58.439783 8468 ts_tablet_manager.cc:1905] T 4045f361f5cd489788b049d84619e9f8 P 85b530590f0848d99c9df9ee4a75e71c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:37:58.439983 8467 ts_tablet_manager.cc:1905] T 4045f361f5cd489788b049d84619e9f8 P 5a55ef951bdf498b898900f2ebe19989: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:37:58.442296 8467 ts_tablet_manager.cc:1918] T 4045f361f5cd489788b049d84619e9f8 P 5a55ef951bdf498b898900f2ebe19989: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:37:58.442386 8467 log.cc:1196] T 4045f361f5cd489788b049d84619e9f8 P 5a55ef951bdf498b898900f2ebe19989: Deleting WAL directory at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/ts-0/wal/wals/4045f361f5cd489788b049d84619e9f8
I20260619 14:37:58.442611 8467 ts_tablet_manager.cc:1939] T 4045f361f5cd489788b049d84619e9f8 P 5a55ef951bdf498b898900f2ebe19989: Deleting consensus metadata
I20260619 14:37:58.442734 8468 ts_tablet_manager.cc:1918] T 4045f361f5cd489788b049d84619e9f8 P 85b530590f0848d99c9df9ee4a75e71c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:37:58.442804 8468 log.cc:1196] T 4045f361f5cd489788b049d84619e9f8 P 85b530590f0848d99c9df9ee4a75e71c: Deleting WAL directory at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/ts-1/wal/wals/4045f361f5cd489788b049d84619e9f8
I20260619 14:37:58.443045 8468 ts_tablet_manager.cc:1939] T 4045f361f5cd489788b049d84619e9f8 P 85b530590f0848d99c9df9ee4a75e71c: Deleting consensus metadata
I20260619 14:37:58.443246 7916 catalog_manager.cc:4841] TS 5a55ef951bdf498b898900f2ebe19989 (127.0.98.1:34993): tablet 4045f361f5cd489788b049d84619e9f8 (table test-table [id=0b4a05ee348241679d2c0691fe53fde6]) successfully deleted
I20260619 14:37:58.443652 8466 ts_tablet_manager.cc:1905] T 4045f361f5cd489788b049d84619e9f8 P 9f164e7dbb0347039e10be0c23084924: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:37:58.446041 8466 ts_tablet_manager.cc:1918] T 4045f361f5cd489788b049d84619e9f8 P 9f164e7dbb0347039e10be0c23084924: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:37:58.446120 8466 log.cc:1196] T 4045f361f5cd489788b049d84619e9f8 P 9f164e7dbb0347039e10be0c23084924: Deleting WAL directory at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/ts-2/wal/wals/4045f361f5cd489788b049d84619e9f8
I20260619 14:37:58.446349 8466 ts_tablet_manager.cc:1939] T 4045f361f5cd489788b049d84619e9f8 P 9f164e7dbb0347039e10be0c23084924: Deleting consensus metadata
I20260619 14:37:58.446985 7917 catalog_manager.cc:4841] TS 9f164e7dbb0347039e10be0c23084924 (127.0.98.3:39321): tablet 4045f361f5cd489788b049d84619e9f8 (table test-table [id=0b4a05ee348241679d2c0691fe53fde6]) successfully deleted
I20260619 14:37:58.447252 7917 catalog_manager.cc:4841] TS 85b530590f0848d99c9df9ee4a75e71c (127.0.98.2:41193): tablet 4045f361f5cd489788b049d84619e9f8 (table test-table [id=0b4a05ee348241679d2c0691fe53fde6]) successfully deleted
Jun 19 14:37:58 dist-test-slave-32hj krb5kdc[7886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879878, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 14:37:58 dist-test-slave-32hj krb5kdc[7886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:37:58 dist-test-slave-32hj krb5kdc[7886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 14:37:58.473278 7980 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:37:58.465426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50336 (local address 127.0.98.62:44517)
0619 14:37:58.465582 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:58.465587 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:58.465748 (+ 161us) server_negotiation.cc:407] Connection header received
0619 14:37:58.465875 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:58.465879 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:58.465934 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:58.466039 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:58.466902 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:58.467699 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:58.469078 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:58.469422 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:58.469538 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:58.473153 (+ 3615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50336: BlockingRecv error: recv got EOF from 127.0.0.1:50336 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 19 14:37:58 dist-test-slave-32hj krb5kdc[7886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879878, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 14:37:58.484268 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:58.475347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50350 (local address 127.0.98.62:44517)
0619 14:37:58.475459 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:58.475463 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:58.475887 (+ 424us) server_negotiation.cc:407] Connection header received
0619 14:37:58.475981 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:58.475984 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:58.476043 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:58.476129 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:37:58.476801 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:58.477571 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:58.478333 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:58.478666 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:58.478778 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:58.481232 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:37:58.481236 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:37:58.481238 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:37:58.481263 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:37:58.482677 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:58.483160 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:58.483164 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:58.483165 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:58.483265 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:37:58.483521 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:37:58.483524 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:37:58.483525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:37:58.483723 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:37:58.483821 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:58.484028 (+ 207us) server_negotiation.cc:299] Negotiation successful
0619 14:37:58.484126 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 14:37:58.486974 7931 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50350:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:37:58.487404 7931 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:37:58.492002 8068 tablet_service.cc:1467] Processing CreateTablet for tablet 21058e1e9b1d4f1d9e8fdf1e91cec570 (DEFAULT_TABLE table=test-table [id=4b2eba5f21944b0f97bf52e138e9781f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:37:58.492347 8068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21058e1e9b1d4f1d9e8fdf1e91cec570. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:58.492319 8348 tablet_service.cc:1467] Processing CreateTablet for tablet 21058e1e9b1d4f1d9e8fdf1e91cec570 (DEFAULT_TABLE table=test-table [id=4b2eba5f21944b0f97bf52e138e9781f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:37:58.492609 8348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21058e1e9b1d4f1d9e8fdf1e91cec570. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:58.494812 8444 tablet_bootstrap.cc:492] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 5a55ef951bdf498b898900f2ebe19989: Bootstrap starting.
I20260619 14:37:58.494997 8208 tablet_service.cc:1467] Processing CreateTablet for tablet 21058e1e9b1d4f1d9e8fdf1e91cec570 (DEFAULT_TABLE table=test-table [id=4b2eba5f21944b0f97bf52e138e9781f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:37:58.495226 8208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21058e1e9b1d4f1d9e8fdf1e91cec570. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:58.495638 8444 tablet_bootstrap.cc:654] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 5a55ef951bdf498b898900f2ebe19989: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:58.496181 8442 tablet_bootstrap.cc:492] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 9f164e7dbb0347039e10be0c23084924: Bootstrap starting.
I20260619 14:37:58.497066 8442 tablet_bootstrap.cc:654] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 9f164e7dbb0347039e10be0c23084924: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:58.497224 8444 tablet_bootstrap.cc:492] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 5a55ef951bdf498b898900f2ebe19989: No bootstrap required, opened a new log
I20260619 14:37:58.497304 8444 ts_tablet_manager.cc:1397] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 5a55ef951bdf498b898900f2ebe19989: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 14:37:58.497781 8443 tablet_bootstrap.cc:492] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 85b530590f0848d99c9df9ee4a75e71c: Bootstrap starting.
I20260619 14:37:58.497720 8444 raft_consensus.cc:348] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 5a55ef951bdf498b898900f2ebe19989 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a55ef951bdf498b898900f2ebe19989" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } } peers { permanent_uuid: "85b530590f0848d99c9df9ee4a75e71c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41193 } } peers { permanent_uuid: "9f164e7dbb0347039e10be0c23084924" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39321 } }
I20260619 14:37:58.497848 8444 raft_consensus.cc:374] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 5a55ef951bdf498b898900f2ebe19989 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:58.497877 8444 raft_consensus.cc:729] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 5a55ef951bdf498b898900f2ebe19989 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a55ef951bdf498b898900f2ebe19989, State: Initialized, Role: FOLLOWER
I20260619 14:37:58.497993 8444 consensus_queue.cc:260] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 5a55ef951bdf498b898900f2ebe19989 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a55ef951bdf498b898900f2ebe19989" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } } peers { permanent_uuid: "85b530590f0848d99c9df9ee4a75e71c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41193 } } peers { permanent_uuid: "9f164e7dbb0347039e10be0c23084924" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39321 } }
I20260619 14:37:58.498188 8442 tablet_bootstrap.cc:492] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 9f164e7dbb0347039e10be0c23084924: No bootstrap required, opened a new log
I20260619 14:37:58.498255 8442 ts_tablet_manager.cc:1397] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 9f164e7dbb0347039e10be0c23084924: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 14:37:58.498257 8444 ts_tablet_manager.cc:1428] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 5a55ef951bdf498b898900f2ebe19989: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 14:37:58.498526 8443 tablet_bootstrap.cc:654] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 85b530590f0848d99c9df9ee4a75e71c: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:58.498636 8442 raft_consensus.cc:348] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 9f164e7dbb0347039e10be0c23084924 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a55ef951bdf498b898900f2ebe19989" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } } peers { permanent_uuid: "85b530590f0848d99c9df9ee4a75e71c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41193 } } peers { permanent_uuid: "9f164e7dbb0347039e10be0c23084924" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39321 } }
I20260619 14:37:58.498723 8442 raft_consensus.cc:374] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 9f164e7dbb0347039e10be0c23084924 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:58.498749 8442 raft_consensus.cc:729] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 9f164e7dbb0347039e10be0c23084924 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f164e7dbb0347039e10be0c23084924, State: Initialized, Role: FOLLOWER
I20260619 14:37:58.498852 8442 consensus_queue.cc:260] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 9f164e7dbb0347039e10be0c23084924 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a55ef951bdf498b898900f2ebe19989" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } } peers { permanent_uuid: "85b530590f0848d99c9df9ee4a75e71c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41193 } } peers { permanent_uuid: "9f164e7dbb0347039e10be0c23084924" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39321 } }
I20260619 14:37:58.499078 8442 ts_tablet_manager.cc:1428] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 9f164e7dbb0347039e10be0c23084924: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:37:58.500087 8443 tablet_bootstrap.cc:492] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 85b530590f0848d99c9df9ee4a75e71c: No bootstrap required, opened a new log
I20260619 14:37:58.500159 8443 ts_tablet_manager.cc:1397] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 85b530590f0848d99c9df9ee4a75e71c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 14:37:58.500524 8443 raft_consensus.cc:348] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 85b530590f0848d99c9df9ee4a75e71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a55ef951bdf498b898900f2ebe19989" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } } peers { permanent_uuid: "85b530590f0848d99c9df9ee4a75e71c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41193 } } peers { permanent_uuid: "9f164e7dbb0347039e10be0c23084924" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39321 } }
I20260619 14:37:58.500623 8443 raft_consensus.cc:374] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 85b530590f0848d99c9df9ee4a75e71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:58.500656 8443 raft_consensus.cc:729] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 85b530590f0848d99c9df9ee4a75e71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85b530590f0848d99c9df9ee4a75e71c, State: Initialized, Role: FOLLOWER
I20260619 14:37:58.500797 8443 consensus_queue.cc:260] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 85b530590f0848d99c9df9ee4a75e71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a55ef951bdf498b898900f2ebe19989" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } } peers { permanent_uuid: "85b530590f0848d99c9df9ee4a75e71c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41193 } } peers { permanent_uuid: "9f164e7dbb0347039e10be0c23084924" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39321 } }
I20260619 14:37:58.501039 8443 ts_tablet_manager.cc:1428] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 85b530590f0848d99c9df9ee4a75e71c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:37:58.822016 8448 raft_consensus.cc:482] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 9f164e7dbb0347039e10be0c23084924 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:37:58.822184 8448 raft_consensus.cc:504] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 9f164e7dbb0347039e10be0c23084924 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a55ef951bdf498b898900f2ebe19989" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } } peers { permanent_uuid: "85b530590f0848d99c9df9ee4a75e71c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41193 } } peers { permanent_uuid: "9f164e7dbb0347039e10be0c23084924" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39321 } }
I20260619 14:37:58.823251 8448 leader_election.cc:288] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 9f164e7dbb0347039e10be0c23084924 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a55ef951bdf498b898900f2ebe19989 (127.0.98.1:34993), 85b530590f0848d99c9df9ee4a75e71c (127.0.98.2:41193)
I20260619 14:37:58.825251 8449 raft_consensus.cc:482] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 85b530590f0848d99c9df9ee4a75e71c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:37:58.825351 8449 raft_consensus.cc:504] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 85b530590f0848d99c9df9ee4a75e71c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a55ef951bdf498b898900f2ebe19989" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } } peers { permanent_uuid: "85b530590f0848d99c9df9ee4a75e71c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41193 } } peers { permanent_uuid: "9f164e7dbb0347039e10be0c23084924" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39321 } }
I20260619 14:37:58.825718 8449 leader_election.cc:288] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 85b530590f0848d99c9df9ee4a75e71c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a55ef951bdf498b898900f2ebe19989 (127.0.98.1:34993), 9f164e7dbb0347039e10be0c23084924 (127.0.98.3:39321)
I20260619 14:37:58.826081 8368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21058e1e9b1d4f1d9e8fdf1e91cec570" candidate_uuid: "85b530590f0848d99c9df9ee4a75e71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f164e7dbb0347039e10be0c23084924" is_pre_election: true
I20260619 14:37:58.826081 8088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21058e1e9b1d4f1d9e8fdf1e91cec570" candidate_uuid: "85b530590f0848d99c9df9ee4a75e71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a55ef951bdf498b898900f2ebe19989" is_pre_election: true
I20260619 14:37:58.826206 8368 raft_consensus.cc:2454] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 9f164e7dbb0347039e10be0c23084924 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85b530590f0848d99c9df9ee4a75e71c in term 0.
I20260619 14:37:58.826206 8088 raft_consensus.cc:2454] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 5a55ef951bdf498b898900f2ebe19989 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85b530590f0848d99c9df9ee4a75e71c in term 0.
I20260619 14:37:58.826500 8160 leader_election.cc:302] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 85b530590f0848d99c9df9ee4a75e71c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a55ef951bdf498b898900f2ebe19989, 85b530590f0848d99c9df9ee4a75e71c; no voters:
I20260619 14:37:58.826745 8449 raft_consensus.cc:2781] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 85b530590f0848d99c9df9ee4a75e71c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:37:58.826794 8449 raft_consensus.cc:482] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 85b530590f0848d99c9df9ee4a75e71c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:37:58.826843 8449 raft_consensus.cc:3037] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 85b530590f0848d99c9df9ee4a75e71c [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:58.827608 8513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:58.823531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34993 (local address 127.0.98.3:50081)
0619 14:37:58.823993 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:58.824004 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:58.824110 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:58.824330 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:58.824333 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:58.824347 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:37:58.824433 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:58.824438 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:58.825225 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:58.825228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:58.826422 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:58.826433 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:58.827200 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:58.827204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:58.827290 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:58.827313 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 14:37:58.827381 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 14:37:58.827449 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"mutex_wait_us":46,"thread_start_us":80,"threads_started":1}
I20260619 14:37:58.827642 8515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:58.823612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50081 (local address 127.0.98.1:34993)
0619 14:37:58.823861 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:58.823865 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:58.824092 (+ 227us) server_negotiation.cc:407] Connection header received
0619 14:37:58.824196 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:58.824199 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:58.824242 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:58.824311 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:37:58.824526 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:58.825107 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:58.826520 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:58.827070 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:58.827150 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:58.827289 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:58.827411 (+ 122us) server_negotiation.cc:299] Negotiation successful
0619 14:37:58.827476 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":76,"threads_started":1}
I20260619 14:37:58.828058 8449 raft_consensus.cc:504] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 85b530590f0848d99c9df9ee4a75e71c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a55ef951bdf498b898900f2ebe19989" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } } peers { permanent_uuid: "85b530590f0848d99c9df9ee4a75e71c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41193 } } peers { permanent_uuid: "9f164e7dbb0347039e10be0c23084924" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39321 } }
I20260619 14:37:58.828285 8088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21058e1e9b1d4f1d9e8fdf1e91cec570" candidate_uuid: "9f164e7dbb0347039e10be0c23084924" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a55ef951bdf498b898900f2ebe19989" is_pre_election: true
I20260619 14:37:58.828415 8449 leader_election.cc:288] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 85b530590f0848d99c9df9ee4a75e71c [CANDIDATE]: Term 1 election: Requested vote from peers 5a55ef951bdf498b898900f2ebe19989 (127.0.98.1:34993), 9f164e7dbb0347039e10be0c23084924 (127.0.98.3:39321)
I20260619 14:37:58.828421 8088 raft_consensus.cc:2454] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 5a55ef951bdf498b898900f2ebe19989 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f164e7dbb0347039e10be0c23084924 in term 0.
I20260619 14:37:58.828642 8088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21058e1e9b1d4f1d9e8fdf1e91cec570" candidate_uuid: "85b530590f0848d99c9df9ee4a75e71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a55ef951bdf498b898900f2ebe19989"
I20260619 14:37:58.828771 8088 raft_consensus.cc:3037] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 5a55ef951bdf498b898900f2ebe19989 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:58.829458 8368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21058e1e9b1d4f1d9e8fdf1e91cec570" candidate_uuid: "85b530590f0848d99c9df9ee4a75e71c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f164e7dbb0347039e10be0c23084924"
I20260619 14:37:58.829490 8300 leader_election.cc:302] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 9f164e7dbb0347039e10be0c23084924 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a55ef951bdf498b898900f2ebe19989, 9f164e7dbb0347039e10be0c23084924; no voters:
I20260619 14:37:58.829572 8368 raft_consensus.cc:3037] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 9f164e7dbb0347039e10be0c23084924 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:58.829942 8088 raft_consensus.cc:2454] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 5a55ef951bdf498b898900f2ebe19989 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85b530590f0848d99c9df9ee4a75e71c in term 1.
I20260619 14:37:58.830255 8160 leader_election.cc:302] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 85b530590f0848d99c9df9ee4a75e71c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a55ef951bdf498b898900f2ebe19989, 85b530590f0848d99c9df9ee4a75e71c; no voters:
I20260619 14:37:58.830364 8514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:58.823531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41193 (local address 127.0.98.3:44155)
0619 14:37:58.823882 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:58.823895 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:58.824012 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:58.824247 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:58.824250 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:58.824264 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:37:58.824358 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:58.824363 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:58.825146 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:58.825149 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:58.829130 (+ 3981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:58.829140 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:58.829949 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:58.829953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:58.830040 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:58.830064 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 14:37:58.830140 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 14:37:58.830212 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":16,"spinlock_wait_cycles":19840,"thread_start_us":89,"threads_started":1}
I20260619 14:37:58.830375 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:58.823612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44155 (local address 127.0.98.2:41193)
0619 14:37:58.823745 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:58.823749 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:58.823983 (+ 234us) server_negotiation.cc:407] Connection header received
0619 14:37:58.824122 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:58.824124 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:58.824167 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:58.824227 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:37:58.824464 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:58.825026 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:58.829243 (+ 4217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:58.829794 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:58.829896 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:58.829956 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:58.830156 (+ 200us) server_negotiation.cc:299] Negotiation successful
0619 14:37:58.830226 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 14:37:58.830622 8449 raft_consensus.cc:2781] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 85b530590f0848d99c9df9ee4a75e71c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:37:58.830681 8368 raft_consensus.cc:2454] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 9f164e7dbb0347039e10be0c23084924 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85b530590f0848d99c9df9ee4a75e71c in term 1.
I20260619 14:37:58.830709 8226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21058e1e9b1d4f1d9e8fdf1e91cec570" candidate_uuid: "9f164e7dbb0347039e10be0c23084924" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85b530590f0848d99c9df9ee4a75e71c" is_pre_election: true
I20260619 14:37:58.830710 8449 raft_consensus.cc:686] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 85b530590f0848d99c9df9ee4a75e71c [term 1 LEADER]: Becoming Leader. State: Replica: 85b530590f0848d99c9df9ee4a75e71c, State: Running, Role: LEADER
I20260619 14:37:58.830798 8448 raft_consensus.cc:2741] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 9f164e7dbb0347039e10be0c23084924 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260619 14:37:58.830914 8449 consensus_queue.cc:237] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 85b530590f0848d99c9df9ee4a75e71c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a55ef951bdf498b898900f2ebe19989" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } } peers { permanent_uuid: "85b530590f0848d99c9df9ee4a75e71c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41193 } } peers { permanent_uuid: "9f164e7dbb0347039e10be0c23084924" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39321 } }
I20260619 14:37:58.832530 7923 catalog_manager.cc:5495] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 85b530590f0848d99c9df9ee4a75e71c reported cstate change: term changed from 0 to 1, leader changed from <none> to 85b530590f0848d99c9df9ee4a75e71c (127.0.98.2). New cstate: current_term: 1 leader_uuid: "85b530590f0848d99c9df9ee4a75e71c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a55ef951bdf498b898900f2ebe19989" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85b530590f0848d99c9df9ee4a75e71c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f164e7dbb0347039e10be0c23084924" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39321 } health_report { overall_health: UNKNOWN } } }
I20260619 14:37:58.867082 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:58.864168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44246 (local address 127.0.98.2:41193)
0619 14:37:58.864317 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:58.864321 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:58.864333 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:37:58.864377 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:58.864379 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:58.864420 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:58.864479 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:37:58.864712 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:58.865387 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:58.866244 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:58.866454 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:58.866559 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:58.866746 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:37:58.866811 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:37:58.866868 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:58.866925 (+ 57us) server_negotiation.cc:299] Negotiation successful
0619 14:37:58.866973 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 14:37:58.868963 8368 raft_consensus.cc:1261] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 9f164e7dbb0347039e10be0c23084924 [term 1 FOLLOWER]: Refusing update from remote peer 85b530590f0848d99c9df9ee4a75e71c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:37:58.868963 8088 raft_consensus.cc:1261] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 5a55ef951bdf498b898900f2ebe19989 [term 1 FOLLOWER]: Refusing update from remote peer 85b530590f0848d99c9df9ee4a75e71c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:37:58.869446 8449 consensus_queue.cc:1035] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 85b530590f0848d99c9df9ee4a75e71c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a55ef951bdf498b898900f2ebe19989" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:37:58.869601 8474 consensus_queue.cc:1035] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 85b530590f0848d99c9df9ee4a75e71c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f164e7dbb0347039e10be0c23084924" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:37:58.876221 7931 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50350:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:37:58.876345 7931 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50350:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:37:58.878125 7931 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d7976811491a44f7add55a6e47dc7e97: Sending DeleteTablet for 3 replicas of tablet 21058e1e9b1d4f1d9e8fdf1e91cec570
I20260619 14:37:58.878599 8068 tablet_service.cc:1514] Processing DeleteTablet for tablet 21058e1e9b1d4f1d9e8fdf1e91cec570 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:37:58 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60504
I20260619 14:37:58.878688 8348 tablet_service.cc:1514] Processing DeleteTablet for tablet 21058e1e9b1d4f1d9e8fdf1e91cec570 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:37:58 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58214
I20260619 14:37:58.878777 8208 tablet_service.cc:1514] Processing DeleteTablet for tablet 21058e1e9b1d4f1d9e8fdf1e91cec570 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:37:58 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50132
I20260619 14:37:58.878841 8466 tablet_replica.cc:330] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 9f164e7dbb0347039e10be0c23084924: stopping tablet replica
I20260619 14:37:58.878927 8467 tablet_replica.cc:330] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 5a55ef951bdf498b898900f2ebe19989: stopping tablet replica
I20260619 14:37:58.878952 8466 raft_consensus.cc:2229] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 9f164e7dbb0347039e10be0c23084924 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:37:58.879025 8467 raft_consensus.cc:2229] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 5a55ef951bdf498b898900f2ebe19989 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:37:58.879065 8466 raft_consensus.cc:2258] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 9f164e7dbb0347039e10be0c23084924 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:37:58.879127 8467 raft_consensus.cc:2258] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 5a55ef951bdf498b898900f2ebe19989 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:37:58.879293 8468 tablet_replica.cc:330] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 85b530590f0848d99c9df9ee4a75e71c: stopping tablet replica
I20260619 14:37:58.879285 7931 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50350:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 14:37:58.879410 8468 raft_consensus.cc:2229] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 85b530590f0848d99c9df9ee4a75e71c [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:37:58.879575 8468 raft_consensus.cc:2258] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 85b530590f0848d99c9df9ee4a75e71c [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 14:37:58.879686 7931 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:37:58.879717 8466 ts_tablet_manager.cc:1905] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 9f164e7dbb0347039e10be0c23084924: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:37:58.880362 8468 ts_tablet_manager.cc:1905] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 85b530590f0848d99c9df9ee4a75e71c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:37:58.882067 8466 ts_tablet_manager.cc:1918] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 9f164e7dbb0347039e10be0c23084924: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:37:58.882145 8466 log.cc:1196] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 9f164e7dbb0347039e10be0c23084924: Deleting WAL directory at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/ts-2/wal/wals/21058e1e9b1d4f1d9e8fdf1e91cec570
I20260619 14:37:58.882367 8466 ts_tablet_manager.cc:1939] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 9f164e7dbb0347039e10be0c23084924: Deleting consensus metadata
I20260619 14:37:58.882761 8468 ts_tablet_manager.cc:1918] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 85b530590f0848d99c9df9ee4a75e71c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:37:58.882834 8468 log.cc:1196] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 85b530590f0848d99c9df9ee4a75e71c: Deleting WAL directory at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/ts-1/wal/wals/21058e1e9b1d4f1d9e8fdf1e91cec570
I20260619 14:37:58.883057 8468 ts_tablet_manager.cc:1939] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 85b530590f0848d99c9df9ee4a75e71c: Deleting consensus metadata
I20260619 14:37:58.883065 7917 catalog_manager.cc:4841] TS 9f164e7dbb0347039e10be0c23084924 (127.0.98.3:39321): tablet 21058e1e9b1d4f1d9e8fdf1e91cec570 (table test-table [id=4b2eba5f21944b0f97bf52e138e9781f]) successfully deleted
I20260619 14:37:58.883734 7917 catalog_manager.cc:4841] TS 85b530590f0848d99c9df9ee4a75e71c (127.0.98.2:41193): tablet 21058e1e9b1d4f1d9e8fdf1e91cec570 (table test-table [id=4b2eba5f21944b0f97bf52e138e9781f]) successfully deleted
I20260619 14:37:58.884070 8467 ts_tablet_manager.cc:1905] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 5a55ef951bdf498b898900f2ebe19989: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:37:58.885756 8068 tablet_service.cc:1467] Processing CreateTablet for tablet 217324e802ea43739ff856418866bbff (DEFAULT_TABLE table=test-table [id=0f1ea0d677d247e682a84a58fa84a837]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:37:58.885792 8208 tablet_service.cc:1467] Processing CreateTablet for tablet 217324e802ea43739ff856418866bbff (DEFAULT_TABLE table=test-table [id=0f1ea0d677d247e682a84a58fa84a837]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:37:58.885993 8208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 217324e802ea43739ff856418866bbff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:58.885993 8068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 217324e802ea43739ff856418866bbff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:58.886229 8467 ts_tablet_manager.cc:1918] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 5a55ef951bdf498b898900f2ebe19989: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:37:58.886281 8467 log.cc:1196] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 5a55ef951bdf498b898900f2ebe19989: Deleting WAL directory at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/ts-0/wal/wals/21058e1e9b1d4f1d9e8fdf1e91cec570
I20260619 14:37:58.886432 8467 ts_tablet_manager.cc:1939] T 21058e1e9b1d4f1d9e8fdf1e91cec570 P 5a55ef951bdf498b898900f2ebe19989: Deleting consensus metadata
I20260619 14:37:58.887086 7916 catalog_manager.cc:4841] TS 5a55ef951bdf498b898900f2ebe19989 (127.0.98.1:34993): tablet 21058e1e9b1d4f1d9e8fdf1e91cec570 (table test-table [id=4b2eba5f21944b0f97bf52e138e9781f]) successfully deleted
I20260619 14:37:58.887969 8443 tablet_bootstrap.cc:492] T 217324e802ea43739ff856418866bbff P 85b530590f0848d99c9df9ee4a75e71c: Bootstrap starting.
I20260619 14:37:58.887969 8444 tablet_bootstrap.cc:492] T 217324e802ea43739ff856418866bbff P 5a55ef951bdf498b898900f2ebe19989: Bootstrap starting.
I20260619 14:37:58.888638 8348 tablet_service.cc:1467] Processing CreateTablet for tablet 217324e802ea43739ff856418866bbff (DEFAULT_TABLE table=test-table [id=0f1ea0d677d247e682a84a58fa84a837]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:37:58.888746 8444 tablet_bootstrap.cc:654] T 217324e802ea43739ff856418866bbff P 5a55ef951bdf498b898900f2ebe19989: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:58.888759 8443 tablet_bootstrap.cc:654] T 217324e802ea43739ff856418866bbff P 85b530590f0848d99c9df9ee4a75e71c: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:58.888890 8348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 217324e802ea43739ff856418866bbff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:37:58.890559 8442 tablet_bootstrap.cc:492] T 217324e802ea43739ff856418866bbff P 9f164e7dbb0347039e10be0c23084924: Bootstrap starting.
I20260619 14:37:58.891244 8442 tablet_bootstrap.cc:654] T 217324e802ea43739ff856418866bbff P 9f164e7dbb0347039e10be0c23084924: Neither blocks nor log segments found. Creating new log.
I20260619 14:37:58.891408 8444 tablet_bootstrap.cc:492] T 217324e802ea43739ff856418866bbff P 5a55ef951bdf498b898900f2ebe19989: No bootstrap required, opened a new log
I20260619 14:37:58.891467 8444 ts_tablet_manager.cc:1397] T 217324e802ea43739ff856418866bbff P 5a55ef951bdf498b898900f2ebe19989: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260619 14:37:58.891681 8443 tablet_bootstrap.cc:492] T 217324e802ea43739ff856418866bbff P 85b530590f0848d99c9df9ee4a75e71c: No bootstrap required, opened a new log
I20260619 14:37:58.891755 8443 ts_tablet_manager.cc:1397] T 217324e802ea43739ff856418866bbff P 85b530590f0848d99c9df9ee4a75e71c: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260619 14:37:58.891822 8444 raft_consensus.cc:348] T 217324e802ea43739ff856418866bbff P 5a55ef951bdf498b898900f2ebe19989 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a55ef951bdf498b898900f2ebe19989" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } } peers { permanent_uuid: "85b530590f0848d99c9df9ee4a75e71c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41193 } } peers { permanent_uuid: "9f164e7dbb0347039e10be0c23084924" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39321 } }
I20260619 14:37:58.891932 8444 raft_consensus.cc:374] T 217324e802ea43739ff856418866bbff P 5a55ef951bdf498b898900f2ebe19989 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:58.891963 8444 raft_consensus.cc:729] T 217324e802ea43739ff856418866bbff P 5a55ef951bdf498b898900f2ebe19989 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a55ef951bdf498b898900f2ebe19989, State: Initialized, Role: FOLLOWER
I20260619 14:37:58.892078 8444 consensus_queue.cc:260] T 217324e802ea43739ff856418866bbff P 5a55ef951bdf498b898900f2ebe19989 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a55ef951bdf498b898900f2ebe19989" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } } peers { permanent_uuid: "85b530590f0848d99c9df9ee4a75e71c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41193 } } peers { permanent_uuid: "9f164e7dbb0347039e10be0c23084924" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39321 } }
I20260619 14:37:58.892060 8443 raft_consensus.cc:348] T 217324e802ea43739ff856418866bbff P 85b530590f0848d99c9df9ee4a75e71c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a55ef951bdf498b898900f2ebe19989" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } } peers { permanent_uuid: "85b530590f0848d99c9df9ee4a75e71c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41193 } } peers { permanent_uuid: "9f164e7dbb0347039e10be0c23084924" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39321 } }
I20260619 14:37:58.892158 8443 raft_consensus.cc:374] T 217324e802ea43739ff856418866bbff P 85b530590f0848d99c9df9ee4a75e71c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:58.892189 8443 raft_consensus.cc:729] T 217324e802ea43739ff856418866bbff P 85b530590f0848d99c9df9ee4a75e71c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85b530590f0848d99c9df9ee4a75e71c, State: Initialized, Role: FOLLOWER
I20260619 14:37:58.892357 8444 ts_tablet_manager.cc:1428] T 217324e802ea43739ff856418866bbff P 5a55ef951bdf498b898900f2ebe19989: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 14:37:58.892329 8443 consensus_queue.cc:260] T 217324e802ea43739ff856418866bbff P 85b530590f0848d99c9df9ee4a75e71c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a55ef951bdf498b898900f2ebe19989" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } } peers { permanent_uuid: "85b530590f0848d99c9df9ee4a75e71c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41193 } } peers { permanent_uuid: "9f164e7dbb0347039e10be0c23084924" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39321 } }
I20260619 14:37:58.892402 8442 tablet_bootstrap.cc:492] T 217324e802ea43739ff856418866bbff P 9f164e7dbb0347039e10be0c23084924: No bootstrap required, opened a new log
I20260619 14:37:58.892473 8442 ts_tablet_manager.cc:1397] T 217324e802ea43739ff856418866bbff P 9f164e7dbb0347039e10be0c23084924: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 14:37:58.892589 8443 ts_tablet_manager.cc:1428] T 217324e802ea43739ff856418866bbff P 85b530590f0848d99c9df9ee4a75e71c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:37:58.892899 8442 raft_consensus.cc:348] T 217324e802ea43739ff856418866bbff P 9f164e7dbb0347039e10be0c23084924 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a55ef951bdf498b898900f2ebe19989" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } } peers { permanent_uuid: "85b530590f0848d99c9df9ee4a75e71c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41193 } } peers { permanent_uuid: "9f164e7dbb0347039e10be0c23084924" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39321 } }
I20260619 14:37:58.892995 8442 raft_consensus.cc:374] T 217324e802ea43739ff856418866bbff P 9f164e7dbb0347039e10be0c23084924 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:37:58.893036 8442 raft_consensus.cc:729] T 217324e802ea43739ff856418866bbff P 9f164e7dbb0347039e10be0c23084924 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f164e7dbb0347039e10be0c23084924, State: Initialized, Role: FOLLOWER
I20260619 14:37:58.893157 8442 consensus_queue.cc:260] T 217324e802ea43739ff856418866bbff P 9f164e7dbb0347039e10be0c23084924 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a55ef951bdf498b898900f2ebe19989" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } } peers { permanent_uuid: "85b530590f0848d99c9df9ee4a75e71c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41193 } } peers { permanent_uuid: "9f164e7dbb0347039e10be0c23084924" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39321 } }
I20260619 14:37:58.893381 8442 ts_tablet_manager.cc:1428] T 217324e802ea43739ff856418866bbff P 9f164e7dbb0347039e10be0c23084924: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:37:58.983466 8450 raft_consensus.cc:482] T 217324e802ea43739ff856418866bbff P 5a55ef951bdf498b898900f2ebe19989 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:37:58.983603 8450 raft_consensus.cc:504] T 217324e802ea43739ff856418866bbff P 5a55ef951bdf498b898900f2ebe19989 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a55ef951bdf498b898900f2ebe19989" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } } peers { permanent_uuid: "85b530590f0848d99c9df9ee4a75e71c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41193 } } peers { permanent_uuid: "9f164e7dbb0347039e10be0c23084924" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39321 } }
I20260619 14:37:58.983983 8450 leader_election.cc:288] T 217324e802ea43739ff856418866bbff P 5a55ef951bdf498b898900f2ebe19989 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85b530590f0848d99c9df9ee4a75e71c (127.0.98.2:41193), 9f164e7dbb0347039e10be0c23084924 (127.0.98.3:39321)
I20260619 14:37:58.984393 8226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217324e802ea43739ff856418866bbff" candidate_uuid: "5a55ef951bdf498b898900f2ebe19989" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85b530590f0848d99c9df9ee4a75e71c" is_pre_election: true
I20260619 14:37:58.984426 8368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217324e802ea43739ff856418866bbff" candidate_uuid: "5a55ef951bdf498b898900f2ebe19989" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f164e7dbb0347039e10be0c23084924" is_pre_election: true
I20260619 14:37:58.984527 8226 raft_consensus.cc:2454] T 217324e802ea43739ff856418866bbff P 85b530590f0848d99c9df9ee4a75e71c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a55ef951bdf498b898900f2ebe19989 in term 0.
I20260619 14:37:58.984563 8368 raft_consensus.cc:2454] T 217324e802ea43739ff856418866bbff P 9f164e7dbb0347039e10be0c23084924 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a55ef951bdf498b898900f2ebe19989 in term 0.
I20260619 14:37:58.984858 8021 leader_election.cc:302] T 217324e802ea43739ff856418866bbff P 5a55ef951bdf498b898900f2ebe19989 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a55ef951bdf498b898900f2ebe19989, 85b530590f0848d99c9df9ee4a75e71c; no voters:
I20260619 14:37:58.985028 8450 raft_consensus.cc:2781] T 217324e802ea43739ff856418866bbff P 5a55ef951bdf498b898900f2ebe19989 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:37:58.985083 8450 raft_consensus.cc:482] T 217324e802ea43739ff856418866bbff P 5a55ef951bdf498b898900f2ebe19989 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:37:58.985119 8450 raft_consensus.cc:3037] T 217324e802ea43739ff856418866bbff P 5a55ef951bdf498b898900f2ebe19989 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:58.986020 8450 raft_consensus.cc:504] T 217324e802ea43739ff856418866bbff P 5a55ef951bdf498b898900f2ebe19989 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a55ef951bdf498b898900f2ebe19989" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } } peers { permanent_uuid: "85b530590f0848d99c9df9ee4a75e71c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41193 } } peers { permanent_uuid: "9f164e7dbb0347039e10be0c23084924" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39321 } }
I20260619 14:37:58.986322 8450 leader_election.cc:288] T 217324e802ea43739ff856418866bbff P 5a55ef951bdf498b898900f2ebe19989 [CANDIDATE]: Term 1 election: Requested vote from peers 85b530590f0848d99c9df9ee4a75e71c (127.0.98.2:41193), 9f164e7dbb0347039e10be0c23084924 (127.0.98.3:39321)
I20260619 14:37:58.986694 8368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217324e802ea43739ff856418866bbff" candidate_uuid: "5a55ef951bdf498b898900f2ebe19989" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f164e7dbb0347039e10be0c23084924"
I20260619 14:37:58.986707 8226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217324e802ea43739ff856418866bbff" candidate_uuid: "5a55ef951bdf498b898900f2ebe19989" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85b530590f0848d99c9df9ee4a75e71c"
I20260619 14:37:58.986791 8368 raft_consensus.cc:3037] T 217324e802ea43739ff856418866bbff P 9f164e7dbb0347039e10be0c23084924 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:58.986814 8226 raft_consensus.cc:3037] T 217324e802ea43739ff856418866bbff P 85b530590f0848d99c9df9ee4a75e71c [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:37:58.987583 8368 raft_consensus.cc:2454] T 217324e802ea43739ff856418866bbff P 9f164e7dbb0347039e10be0c23084924 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a55ef951bdf498b898900f2ebe19989 in term 1.
I20260619 14:37:58.987687 8226 raft_consensus.cc:2454] T 217324e802ea43739ff856418866bbff P 85b530590f0848d99c9df9ee4a75e71c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a55ef951bdf498b898900f2ebe19989 in term 1.
I20260619 14:37:58.987807 8021 leader_election.cc:302] T 217324e802ea43739ff856418866bbff P 5a55ef951bdf498b898900f2ebe19989 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a55ef951bdf498b898900f2ebe19989, 9f164e7dbb0347039e10be0c23084924; no voters:
I20260619 14:37:58.987957 8450 raft_consensus.cc:2781] T 217324e802ea43739ff856418866bbff P 5a55ef951bdf498b898900f2ebe19989 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:37:58.988124 8450 raft_consensus.cc:686] T 217324e802ea43739ff856418866bbff P 5a55ef951bdf498b898900f2ebe19989 [term 1 LEADER]: Becoming Leader. State: Replica: 5a55ef951bdf498b898900f2ebe19989, State: Running, Role: LEADER
I20260619 14:37:58.988255 8450 consensus_queue.cc:237] T 217324e802ea43739ff856418866bbff P 5a55ef951bdf498b898900f2ebe19989 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a55ef951bdf498b898900f2ebe19989" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } } peers { permanent_uuid: "85b530590f0848d99c9df9ee4a75e71c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41193 } } peers { permanent_uuid: "9f164e7dbb0347039e10be0c23084924" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39321 } }
I20260619 14:37:58.989727 7931 catalog_manager.cc:5495] T 217324e802ea43739ff856418866bbff P 5a55ef951bdf498b898900f2ebe19989 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a55ef951bdf498b898900f2ebe19989 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "5a55ef951bdf498b898900f2ebe19989" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a55ef951bdf498b898900f2ebe19989" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85b530590f0848d99c9df9ee4a75e71c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f164e7dbb0347039e10be0c23084924" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39321 } health_report { overall_health: UNKNOWN } } }
I20260619 14:37:59.021222 8515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:59.018578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41738 (local address 127.0.98.1:34993)
0619 14:37:59.018705 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:59.018708 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:59.018734 (+ 26us) server_negotiation.cc:407] Connection header received
0619 14:37:59.018848 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:59.018850 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:59.018892 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:59.018950 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:37:59.019112 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:59.019594 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:59.020315 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:59.020511 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:59.020576 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:59.020872 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:37:59.020936 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:37:59.020981 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:59.021071 (+ 90us) server_negotiation.cc:299] Negotiation successful
0619 14:37:59.021116 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 14:37:59.030009 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:59.027370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35147 (local address 127.0.98.2:41193)
0619 14:37:59.027473 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:59.027477 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:59.027529 (+ 52us) server_negotiation.cc:407] Connection header received
0619 14:37:59.027640 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:59.027642 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:59.027682 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:59.027744 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:37:59.027917 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:59.028403 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:59.029232 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:59.029431 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:59.029499 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:59.029695 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:37:59.029758 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:37:59.029805 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:59.029863 (+ 58us) server_negotiation.cc:299] Negotiation successful
0619 14:37:59.029903 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 14:37:59.030033 8527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:59.027158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41193 (local address 127.0.98.1:35147)
0619 14:37:59.027455 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:59.027468 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:59.027564 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:59.027757 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:59.027759 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:59.027765 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:37:59.027847 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:59.027851 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:59.028481 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:59.028483 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:59.029128 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:59.029136 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:59.029505 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:59.029507 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:59.029562 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:59.029601 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:37:59.029811 (+ 210us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:37:59.029814 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 14:37:59.029855 (+ 41us) client_negotiation.cc:240] Negotiation successful
0619 14:37:59.029917 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":119,"threads_started":1}
I20260619 14:37:59.034986 8368 raft_consensus.cc:1261] T 217324e802ea43739ff856418866bbff P 9f164e7dbb0347039e10be0c23084924 [term 1 FOLLOWER]: Refusing update from remote peer 5a55ef951bdf498b898900f2ebe19989: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:37:59.035053 8226 raft_consensus.cc:1261] T 217324e802ea43739ff856418866bbff P 85b530590f0848d99c9df9ee4a75e71c [term 1 FOLLOWER]: Refusing update from remote peer 5a55ef951bdf498b898900f2ebe19989: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:37:59.035395 8450 consensus_queue.cc:1035] T 217324e802ea43739ff856418866bbff P 5a55ef951bdf498b898900f2ebe19989 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85b530590f0848d99c9df9ee4a75e71c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:37:59.035537 8523 consensus_queue.cc:1035] T 217324e802ea43739ff856418866bbff P 5a55ef951bdf498b898900f2ebe19989 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f164e7dbb0347039e10be0c23084924" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:37:59.050290 8515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:59.047374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55523 (local address 127.0.98.1:34993)
0619 14:37:59.047482 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 14:37:59.047486 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:37:59.047623 (+ 137us) server_negotiation.cc:407] Connection header received
0619 14:37:59.047729 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:37:59.047732 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:37:59.047770 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:37:59.047832 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:37:59.048050 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:59.048541 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:59.049361 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:37:59.049572 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:37:59.049651 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:59.049877 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:37:59.049955 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:37:59.050023 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 14:37:59.050096 (+ 73us) server_negotiation.cc:299] Negotiation successful
0619 14:37:59.050148 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 14:37:59.050309 8532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:59.047320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34993 (local address 127.0.98.2:55523)
0619 14:37:59.047535 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:59.047549 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:37:59.047644 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:37:59.047852 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:37:59.047855 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:37:59.047863 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:37:59.047959 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:59.047965 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:59.048631 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:59.048635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:59.049253 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:37:59.049261 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:37:59.049660 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:37:59.049663 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:37:59.049748 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:37:59.049776 (+ 28us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:37:59.050030 (+ 254us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:37:59.050035 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 14:37:59.050082 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 14:37:59.050165 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":75,"threads_started":1}
I20260619 14:37:59.069872 7930 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50350:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:37:59.070024 7930 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50350:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:37:59.071930 7930 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d7976811491a44f7add55a6e47dc7e97: Sending DeleteTablet for 3 replicas of tablet 217324e802ea43739ff856418866bbff
I20260619 14:37:59.072499 8068 tablet_service.cc:1514] Processing DeleteTablet for tablet 217324e802ea43739ff856418866bbff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:37:59 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60504
I20260619 14:37:59.072593 8208 tablet_service.cc:1514] Processing DeleteTablet for tablet 217324e802ea43739ff856418866bbff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:37:59 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50132
I20260619 14:37:59.072623 8348 tablet_service.cc:1514] Processing DeleteTablet for tablet 217324e802ea43739ff856418866bbff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:37:59 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58214
I20260619 14:37:59.072706 8467 tablet_replica.cc:330] T 217324e802ea43739ff856418866bbff P 5a55ef951bdf498b898900f2ebe19989: stopping tablet replica
I20260619 14:37:59.072801 8468 tablet_replica.cc:330] T 217324e802ea43739ff856418866bbff P 85b530590f0848d99c9df9ee4a75e71c: stopping tablet replica
I20260619 14:37:59.072841 8467 raft_consensus.cc:2229] T 217324e802ea43739ff856418866bbff P 5a55ef951bdf498b898900f2ebe19989 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:37:59.072916 8468 raft_consensus.cc:2229] T 217324e802ea43739ff856418866bbff P 85b530590f0848d99c9df9ee4a75e71c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:37:59.073031 8468 raft_consensus.cc:2258] T 217324e802ea43739ff856418866bbff P 85b530590f0848d99c9df9ee4a75e71c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:37:59.073036 8467 raft_consensus.cc:2258] T 217324e802ea43739ff856418866bbff P 5a55ef951bdf498b898900f2ebe19989 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:37:59.073096 8466 tablet_replica.cc:330] T 217324e802ea43739ff856418866bbff P 9f164e7dbb0347039e10be0c23084924: stopping tablet replica
I20260619 14:37:59.073204 8466 raft_consensus.cc:2229] T 217324e802ea43739ff856418866bbff P 9f164e7dbb0347039e10be0c23084924 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:37:59.073311 8466 raft_consensus.cc:2258] T 217324e802ea43739ff856418866bbff P 9f164e7dbb0347039e10be0c23084924 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:37:59.073423 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 8003
I20260619 14:37:59.073714 8468 ts_tablet_manager.cc:1905] T 217324e802ea43739ff856418866bbff P 85b530590f0848d99c9df9ee4a75e71c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:37:59.073725 8467 ts_tablet_manager.cc:1905] T 217324e802ea43739ff856418866bbff P 5a55ef951bdf498b898900f2ebe19989: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:37:59.075863 8466 ts_tablet_manager.cc:1905] T 217324e802ea43739ff856418866bbff P 9f164e7dbb0347039e10be0c23084924: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:37:59.076328 8468 ts_tablet_manager.cc:1918] T 217324e802ea43739ff856418866bbff P 85b530590f0848d99c9df9ee4a75e71c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:37:59.076406 8468 log.cc:1196] T 217324e802ea43739ff856418866bbff P 85b530590f0848d99c9df9ee4a75e71c: Deleting WAL directory at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/ts-1/wal/wals/217324e802ea43739ff856418866bbff
I20260619 14:37:59.076651 8468 ts_tablet_manager.cc:1939] T 217324e802ea43739ff856418866bbff P 85b530590f0848d99c9df9ee4a75e71c: Deleting consensus metadata
I20260619 14:37:59.077370 7917 catalog_manager.cc:4841] TS 85b530590f0848d99c9df9ee4a75e71c (127.0.98.2:41193): tablet 217324e802ea43739ff856418866bbff (table test-table [id=0f1ea0d677d247e682a84a58fa84a837]) successfully deleted
I20260619 14:37:59.077919 8466 ts_tablet_manager.cc:1918] T 217324e802ea43739ff856418866bbff P 9f164e7dbb0347039e10be0c23084924: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:37:59.077984 8466 log.cc:1196] T 217324e802ea43739ff856418866bbff P 9f164e7dbb0347039e10be0c23084924: Deleting WAL directory at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781879837648036-392-0/minicluster-data/ts-2/wal/wals/217324e802ea43739ff856418866bbff
I20260619 14:37:59.078164 8466 ts_tablet_manager.cc:1939] T 217324e802ea43739ff856418866bbff P 9f164e7dbb0347039e10be0c23084924: Deleting consensus metadata
I20260619 14:37:59.078774 7917 catalog_manager.cc:4841] TS 9f164e7dbb0347039e10be0c23084924 (127.0.98.3:39321): tablet 217324e802ea43739ff856418866bbff (table test-table [id=0f1ea0d677d247e682a84a58fa84a837]) successfully deleted
W20260619 14:37:59.080886 7916 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34993 (error 108)
I20260619 14:37:59.081101 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 8143
I20260619 14:37:59.082948 8535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:37:59.082559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34993 (local address 127.0.0.1:41744)
0619 14:37:59.082785 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 14:37:59.082859 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":63,"threads_started":1}
W20260619 14:37:59.083065 7916 catalog_manager.cc:4584] TS 5a55ef951bdf498b898900f2ebe19989 (127.0.98.1:34993): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 217324e802ea43739ff856418866bbff: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34993: connect: Connection refused (error 111)
I20260619 14:37:59.087415 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 8283
I20260619 14:37:59.092475 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 7899
2026-06-19T14:37:59Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3367 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781879837648036-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:37:59 dist-test-slave-32hj krb5kdc[8538](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:37:59 dist-test-slave-32hj krb5kdc[8538](info): set up 2 sockets
Jun 19 14:37:59 dist-test-slave-32hj krb5kdc[8538](info): commencing operation
krb5kdc: starting...
W20260619 14:38:00.045118 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781879837648036-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781879837648036-392-0/krb5kdc/test-user.keytab.
Jun 19 14:38:00 dist-test-slave-32hj krb5kdc[8538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:38:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:38:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:38:00.161695 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781879837648036-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781879837648036-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781879837648036-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37671
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37671
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:38:00.269872 8551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:38:00.270171 8551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:38:00.270215 8551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:38:00.274297 8551 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:38:00.274372 8551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:38:00.274394 8551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:38:00.274412 8551 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:38:00.274438 8551 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:38:00.279907 8551 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781879837648036-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781879837648036-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37671
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781879837648036-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:38:00.281024 8551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:38:00.282047 8551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:38:00.290458 8557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:38:00.290472 8556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:38:00.290584 8551 server_base.cc:1034] running on GCE node
W20260619 14:38:00.290450 8559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:38:00.291360 8551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:38:00.292514 8551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:38:00.293696 8551 hybrid_clock.cc:648] HybridClock initialized: now 1781879880293664 us; error 48 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-19T14:38:00Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1207 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:38:00 dist-test-slave-32hj krb5kdc[8564](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:38:00 dist-test-slave-32hj krb5kdc[8564](info): set up 2 sockets
Jun 19 14:38:00 dist-test-slave-32hj krb5kdc[8564](info): commencing operation
krb5kdc: starting...
W20260619 14:38:01.265108 392 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/krb5kdc/test-user.keytab.
Jun 19 14:38:01 dist-test-slave-32hj krb5kdc[8564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:38:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:38:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:38:01.383147 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34179
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34179
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:38:01.489430 8577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:38:01.489696 8577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:38:01.489734 8577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:38:01.493682 8577 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:38:01.493739 8577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:38:01.493757 8577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:38:01.493772 8577 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:38:01.493785 8577 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:38:01.499027 8577 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34179
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:38:01.500037 8577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:38:01.500995 8577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:38:01.509006 8583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:38:01.509058 8585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:38:01.509001 8582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:38:01.509196 8577 server_base.cc:1034] running on GCE node
I20260619 14:38:01.509752 8577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:38:01.510812 8577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:38:01.511963 8577 hybrid_clock.cc:648] HybridClock initialized: now 1781879881511949 us; error 31 us; skew 500 ppm
Jun 19 14:38:01 dist-test-slave-32hj krb5kdc[8564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:38:01.514791 8577 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:38:01.515868 8577 webserver.cc:460] Webserver started at http://127.0.98.62:42997/ using document root <none> and password file <none>
I20260619 14:38:01.516536 8577 fs_manager.cc:359] Metadata directory not provided
I20260619 14:38:01.516621 8577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:38:01.516882 8577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:38:01.519091 8577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/master-0/data/instance:
uuid: "f5e1fe3efbd848e6b60728f62f513fee"
format_stamp: "Formatted at 2026-06-19 14:38:01 on dist-test-slave-32hj"
server_key: "59578fd1aedd06b5dcaf5441efd0a577"
server_key_iv: "28e6002303b5d00de9e0e6cfc84d7f9c"
server_key_version: "encryptionkey@0"
I20260619 14:38:01.519742 8577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/master-0/wal/instance:
uuid: "f5e1fe3efbd848e6b60728f62f513fee"
format_stamp: "Formatted at 2026-06-19 14:38:01 on dist-test-slave-32hj"
server_key: "59578fd1aedd06b5dcaf5441efd0a577"
server_key_iv: "28e6002303b5d00de9e0e6cfc84d7f9c"
server_key_version: "encryptionkey@0"
I20260619 14:38:01.523342 8577 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 14:38:01.525727 8592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:38:01.526983 8577 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:38:01.527087 8577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/master-0/wal
uuid: "f5e1fe3efbd848e6b60728f62f513fee"
format_stamp: "Formatted at 2026-06-19 14:38:01 on dist-test-slave-32hj"
server_key: "59578fd1aedd06b5dcaf5441efd0a577"
server_key_iv: "28e6002303b5d00de9e0e6cfc84d7f9c"
server_key_version: "encryptionkey@0"
I20260619 14:38:01.527185 8577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:38:01.537886 8577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:38:01.540607 8577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:38:01.540823 8577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:38:01.547323 8577 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34179
I20260619 14:38:01.547372 8644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34179 every 8 connection(s)
I20260619 14:38:01.548218 8577 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:38:01.548851 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 8577
I20260619 14:38:01.548971 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/master-0/wal/instance
I20260619 14:38:01.549248 392 external_mini_cluster.cc:1428] Setting key 737da5fb84f72c9ff6857e6bc5fa8f5d
I20260619 14:38:01.551015 8645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:38:01.557121 8645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5e1fe3efbd848e6b60728f62f513fee: Bootstrap starting.
Jun 19 14:38:01 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:38:01.559520 8645 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5e1fe3efbd848e6b60728f62f513fee: Neither blocks nor log segments found. Creating new log.
I20260619 14:38:01.560230 8645 log.cc:824] T 00000000000000000000000000000000 P f5e1fe3efbd848e6b60728f62f513fee: Log is configured to *not* fsync() on all Append() calls
I20260619 14:38:01.562276 8645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5e1fe3efbd848e6b60728f62f513fee: No bootstrap required, opened a new log
I20260619 14:38:01.562855 8648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:01.550550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38632 (local address 127.0.98.62:34179)
0619 14:38:01.550935 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:01.550950 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:01.550982 (+ 32us) server_negotiation.cc:407] Connection header received
0619 14:38:01.551598 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:01.551613 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:01.551932 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:01.552209 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:01.553058 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:01.554028 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:01.555139 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:01.555580 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:01.555704 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:01.558375 (+ 2671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:38:01.558382 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:38:01.558397 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:38:01.558428 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:38:01.559987 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:38:01.560430 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:38:01.560435 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:38:01.560439 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:38:01.560547 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:38:01.560838 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:38:01.560841 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:38:01.560842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:38:01.561264 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:38:01.561409 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 14:38:01.561666 (+ 257us) server_negotiation.cc:299] Negotiation successful
0619 14:38:01.561927 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":109,"threads_started":1}
I20260619 14:38:01.565194 8645 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5e1fe3efbd848e6b60728f62f513fee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5e1fe3efbd848e6b60728f62f513fee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34179 } }
I20260619 14:38:01.565400 8645 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5e1fe3efbd848e6b60728f62f513fee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:38:01.565449 8645 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5e1fe3efbd848e6b60728f62f513fee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5e1fe3efbd848e6b60728f62f513fee, State: Initialized, Role: FOLLOWER
I20260619 14:38:01.565817 8645 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5e1fe3efbd848e6b60728f62f513fee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5e1fe3efbd848e6b60728f62f513fee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34179 } }
I20260619 14:38:01.565928 8645 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5e1fe3efbd848e6b60728f62f513fee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:38:01.565989 8645 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5e1fe3efbd848e6b60728f62f513fee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:38:01.566073 8645 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5e1fe3efbd848e6b60728f62f513fee [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:38:01.566982 8645 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5e1fe3efbd848e6b60728f62f513fee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5e1fe3efbd848e6b60728f62f513fee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34179 } }
I20260619 14:38:01.567265 8645 leader_election.cc:302] T 00000000000000000000000000000000 P f5e1fe3efbd848e6b60728f62f513fee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5e1fe3efbd848e6b60728f62f513fee; no voters:
I20260619 14:38:01.567493 8645 leader_election.cc:288] T 00000000000000000000000000000000 P f5e1fe3efbd848e6b60728f62f513fee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:38:01.567621 8650 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5e1fe3efbd848e6b60728f62f513fee [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:38:01.567827 8650 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5e1fe3efbd848e6b60728f62f513fee [term 1 LEADER]: Becoming Leader. State: Replica: f5e1fe3efbd848e6b60728f62f513fee, State: Running, Role: LEADER
I20260619 14:38:01.568101 8650 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5e1fe3efbd848e6b60728f62f513fee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5e1fe3efbd848e6b60728f62f513fee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34179 } }
I20260619 14:38:01.568562 8645 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5e1fe3efbd848e6b60728f62f513fee [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:38:01.569697 8651 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5e1fe3efbd848e6b60728f62f513fee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5e1fe3efbd848e6b60728f62f513fee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5e1fe3efbd848e6b60728f62f513fee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34179 } } }
I20260619 14:38:01.569720 8652 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5e1fe3efbd848e6b60728f62f513fee [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5e1fe3efbd848e6b60728f62f513fee. Latest consensus state: current_term: 1 leader_uuid: "f5e1fe3efbd848e6b60728f62f513fee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5e1fe3efbd848e6b60728f62f513fee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34179 } } }
I20260619 14:38:01.569826 8651 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5e1fe3efbd848e6b60728f62f513fee [sys.catalog]: This master's current role is: LEADER
I20260619 14:38:01.569831 8652 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5e1fe3efbd848e6b60728f62f513fee [sys.catalog]: This master's current role is: LEADER
I20260619 14:38:01.570173 8659 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:38:01.572754 8659 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:38:01.577862 8659 catalog_manager.cc:1347] Generated new cluster ID: 2a063f592c5045b49b7ffbb3ff82472f
I20260619 14:38:01.577916 8659 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:38:01.589110 8659 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:38:01.590255 8659 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:38:01.602303 8659 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f5e1fe3efbd848e6b60728f62f513fee: Generated new TSK 0
I20260619 14:38:01.603075 8659 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 14:38:01.637287 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34179
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:38:01.750337 8671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:38:01.750588 8671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:38:01.750638 8671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:38:01.754644 8671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:38:01.754701 8671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:38:01.754758 8671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:38:01.760017 8671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:38:01.761049 8671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:38:01.762012 8671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:38:01.771137 8676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:38:01.771157 8671 server_base.cc:1034] running on GCE node
W20260619 14:38:01.771179 8677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:38:01.771214 8679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:38:01.771747 8671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:38:01.772362 8671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:38:01.773522 8671 hybrid_clock.cc:648] HybridClock initialized: now 1781879881773503 us; error 36 us; skew 500 ppm
Jun 19 14:38:01 dist-test-slave-32hj krb5kdc[8564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:38:01.776177 8671 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:38:01.777064 8671 webserver.cc:460] Webserver started at http://127.0.98.1:34195/ using document root <none> and password file <none>
I20260619 14:38:01.777581 8671 fs_manager.cc:359] Metadata directory not provided
I20260619 14:38:01.777660 8671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:38:01.777863 8671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:38:01.779852 8671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-0/data/instance:
uuid: "6f8a85683eb74926b43ae1d5ef6337f1"
format_stamp: "Formatted at 2026-06-19 14:38:01 on dist-test-slave-32hj"
server_key: "afb2b09909732ac09d9a65058c3b5a68"
server_key_iv: "f366239fc8d873f56a3a2b6781c4e969"
server_key_version: "encryptionkey@0"
I20260619 14:38:01.780382 8671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6f8a85683eb74926b43ae1d5ef6337f1"
format_stamp: "Formatted at 2026-06-19 14:38:01 on dist-test-slave-32hj"
server_key: "afb2b09909732ac09d9a65058c3b5a68"
server_key_iv: "f366239fc8d873f56a3a2b6781c4e969"
server_key_version: "encryptionkey@0"
I20260619 14:38:01.784050 8671 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 14:38:01.786466 8686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:38:01.787698 8671 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:38:01.787803 8671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-0/wal
uuid: "6f8a85683eb74926b43ae1d5ef6337f1"
format_stamp: "Formatted at 2026-06-19 14:38:01 on dist-test-slave-32hj"
server_key: "afb2b09909732ac09d9a65058c3b5a68"
server_key_iv: "f366239fc8d873f56a3a2b6781c4e969"
server_key_version: "encryptionkey@0"
I20260619 14:38:01.787899 8671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:38:01.811684 8671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:38:01.814955 8671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:38:01.815142 8671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:38:01.815752 8671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:38:01.816572 8671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:38:01.816632 8671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:38:01.816686 8671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:38:01.816712 8671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:38:01.824690 8671 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41541
I20260619 14:38:01.824771 8799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41541 every 8 connection(s)
I20260619 14:38:01.825619 8671 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:38:01.827356 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 8671
I20260619 14:38:01.827484 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:38:01.827749 392 external_mini_cluster.cc:1428] Setting key 85989ab3235900eab7b04f2fa6117042
Jun 19 14:38:01 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 14:38:01.840369 8648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:01.827361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41465 (local address 127.0.98.62:34179)
0619 14:38:01.827510 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:01.827515 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:01.828370 (+ 855us) server_negotiation.cc:407] Connection header received
0619 14:38:01.830000 (+ 1630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:01.830003 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:01.830044 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:01.830111 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:01.831507 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:01.832148 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:01.833789 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:01.834014 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:01.834095 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:01.836850 (+ 2755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:38:01.836854 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:38:01.836857 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:38:01.836884 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:38:01.838187 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:38:01.838619 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:38:01.838621 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:38:01.838623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:38:01.838690 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:38:01.839111 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:38:01.839114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:38:01.839116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:38:01.839354 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:38:01.839491 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 14:38:01.840065 (+ 574us) server_negotiation.cc:299] Negotiation successful
0619 14:38:01.840209 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 14:38:01.841174 8802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:01.827740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34179 (local address 127.0.98.1:41465)
0619 14:38:01.828213 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:01.828260 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:01.829786 (+ 1526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:01.830291 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:01.830300 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:01.830815 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:38:01.831376 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:01.831391 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:01.832283 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:01.832287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:01.833675 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:01.833686 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:01.834170 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:01.834175 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:01.834379 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:01.834962 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:38:01.834982 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:38:01.836633 (+ 1651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:38:01.838284 (+ 1651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:38:01.838289 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:38:01.838301 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:38:01.838539 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:38:01.838780 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:38:01.838783 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:38:01.838785 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:38:01.838984 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:38:01.839560 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:38:01.839567 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:38:01.839814 (+ 247us) client_negotiation.cc:769] Sending connection context
0619 14:38:01.839965 (+ 151us) client_negotiation.cc:240] Negotiation successful
0619 14:38:01.840225 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":123,"threads_started":1}
I20260619 14:38:01.842451 8800 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34179
I20260619 14:38:01.842696 8800 heartbeater.cc:464] Registering TS with master...
I20260619 14:38:01.843264 8800 heartbeater.cc:511] Master 127.0.98.62:34179 requested a full tablet report, sending...
I20260619 14:38:01.844777 8609 ts_manager.cc:194] Registered new tserver with Master: 6f8a85683eb74926b43ae1d5ef6337f1 (127.0.98.1:41541)
I20260619 14:38:01.845876 8609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41465
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 14:38:01.855473 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34179
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:38:01.960278 8805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:38:01.960533 8805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:38:01.960587 8805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:38:01.964619 8805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:38:01.964679 8805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:38:01.964773 8805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:38:01.969960 8805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:38:01.970963 8805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:38:01.971912 8805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:38:01.980926 8811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:38:01.980903 8813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:38:01.980901 8810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:38:01.980978 8805 server_base.cc:1034] running on GCE node
I20260619 14:38:01.981568 8805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:38:01.982179 8805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:38:01.983328 8805 hybrid_clock.cc:648] HybridClock initialized: now 1781879881983307 us; error 33 us; skew 500 ppm
Jun 19 14:38:01 dist-test-slave-32hj krb5kdc[8564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:38:01.986053 8805 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:38:01.986913 8805 webserver.cc:460] Webserver started at http://127.0.98.2:46673/ using document root <none> and password file <none>
I20260619 14:38:01.987434 8805 fs_manager.cc:359] Metadata directory not provided
I20260619 14:38:01.987493 8805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:38:01.987689 8805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:38:01.989768 8805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-1/data/instance:
uuid: "722677b14e094f2d93709caa5aac50b5"
format_stamp: "Formatted at 2026-06-19 14:38:01 on dist-test-slave-32hj"
server_key: "27fe22de353df4994616cf1275755ef1"
server_key_iv: "048b6f2ba3252320a1a9edac5b8ba18c"
server_key_version: "encryptionkey@0"
I20260619 14:38:01.990312 8805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-1/wal/instance:
uuid: "722677b14e094f2d93709caa5aac50b5"
format_stamp: "Formatted at 2026-06-19 14:38:01 on dist-test-slave-32hj"
server_key: "27fe22de353df4994616cf1275755ef1"
server_key_iv: "048b6f2ba3252320a1a9edac5b8ba18c"
server_key_version: "encryptionkey@0"
I20260619 14:38:01.994155 8805 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:38:01.996620 8820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:38:01.997818 8805 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:38:01.997915 8805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-1/wal
uuid: "722677b14e094f2d93709caa5aac50b5"
format_stamp: "Formatted at 2026-06-19 14:38:01 on dist-test-slave-32hj"
server_key: "27fe22de353df4994616cf1275755ef1"
server_key_iv: "048b6f2ba3252320a1a9edac5b8ba18c"
server_key_version: "encryptionkey@0"
I20260619 14:38:01.998019 8805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:38:02.036800 8805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:38:02.039705 8805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:38:02.039855 8805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:38:02.040441 8805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:38:02.041294 8805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:38:02.041342 8805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:38:02.041393 8805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:38:02.041422 8805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:38:02.049642 8805 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35537
I20260619 14:38:02.049662 8933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35537 every 8 connection(s)
I20260619 14:38:02.050513 8805 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:38:02.051540 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 8805
I20260619 14:38:02.051685 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:38:02.051986 392 external_mini_cluster.cc:1428] Setting key 0dd408f41f17deb36c3ce5385f5f74db
Jun 19 14:38:02 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 14:38:02.065841 8648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:02.052334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60753 (local address 127.0.98.62:34179)
0619 14:38:02.052483 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:02.052488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:02.053496 (+ 1008us) server_negotiation.cc:407] Connection header received
0619 14:38:02.055199 (+ 1703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:02.055202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:02.055246 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:02.055308 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:02.056817 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:02.057547 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:02.059291 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:02.059505 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:02.059602 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:02.062329 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:38:02.062332 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:38:02.062334 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:38:02.062358 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:38:02.063723 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:38:02.064332 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:38:02.064335 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:38:02.064337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:38:02.064426 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:38:02.064801 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:38:02.064805 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:38:02.064806 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:38:02.065015 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:38:02.065107 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 14:38:02.065614 (+ 507us) server_negotiation.cc:299] Negotiation successful
0619 14:38:02.065719 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 14:38:02.066916 8936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:02.052822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34179 (local address 127.0.98.2:60753)
0619 14:38:02.053336 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:02.053395 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:02.054959 (+ 1564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:02.055503 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:02.055512 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:02.055993 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:38:02.056662 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:02.056682 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:02.057699 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:02.057703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:02.059155 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:02.059167 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:02.059663 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:02.059668 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:02.059871 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:02.060478 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:38:02.060500 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:38:02.062168 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:38:02.063816 (+ 1648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:38:02.063822 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:38:02.063834 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:38:02.064232 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:38:02.064527 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:38:02.064529 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:38:02.064531 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:38:02.064672 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:38:02.065100 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:38:02.065105 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:38:02.065350 (+ 245us) client_negotiation.cc:769] Sending connection context
0619 14:38:02.065537 (+ 187us) client_negotiation.cc:240] Negotiation successful
0619 14:38:02.065887 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":124,"threads_started":1}
I20260619 14:38:02.068037 8934 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34179
I20260619 14:38:02.068286 8934 heartbeater.cc:464] Registering TS with master...
I20260619 14:38:02.068758 8934 heartbeater.cc:511] Master 127.0.98.62:34179 requested a full tablet report, sending...
I20260619 14:38:02.069767 8609 ts_manager.cc:194] Registered new tserver with Master: 722677b14e094f2d93709caa5aac50b5 (127.0.98.2:35537)
I20260619 14:38:02.070349 8609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60753
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 14:38:02.079074 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34179
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:38:02.186159 8939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:38:02.186424 8939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:38:02.186494 8939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:38:02.190557 8939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:38:02.190615 8939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:38:02.190693 8939 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:38:02.195894 8939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:38:02.196928 8939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:38:02.197888 8939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:38:02.206822 8944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:38:02.206831 8945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:38:02.206869 8947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:38:02.207170 8939 server_base.cc:1034] running on GCE node
I20260619 14:38:02.207513 8939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:38:02.208112 8939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:38:02.209260 8939 hybrid_clock.cc:648] HybridClock initialized: now 1781879882209234 us; error 43 us; skew 500 ppm
Jun 19 14:38:02 dist-test-slave-32hj krb5kdc[8564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879882, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:38:02.211768 8939 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:38:02.212606 8939 webserver.cc:460] Webserver started at http://127.0.98.3:45249/ using document root <none> and password file <none>
I20260619 14:38:02.213176 8939 fs_manager.cc:359] Metadata directory not provided
I20260619 14:38:02.213241 8939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:38:02.213436 8939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:38:02.215411 8939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-2/data/instance:
uuid: "24f77571e2ef4fb39c11addd8d6d767e"
format_stamp: "Formatted at 2026-06-19 14:38:02 on dist-test-slave-32hj"
server_key: "fff3e3c796ea8c95d2aadcfc833ce91c"
server_key_iv: "8b517e9439b2a32a8fc776f87576733f"
server_key_version: "encryptionkey@0"
I20260619 14:38:02.215946 8939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-2/wal/instance:
uuid: "24f77571e2ef4fb39c11addd8d6d767e"
format_stamp: "Formatted at 2026-06-19 14:38:02 on dist-test-slave-32hj"
server_key: "fff3e3c796ea8c95d2aadcfc833ce91c"
server_key_iv: "8b517e9439b2a32a8fc776f87576733f"
server_key_version: "encryptionkey@0"
I20260619 14:38:02.219544 8939 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 14:38:02.221910 8954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:38:02.223129 8939 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:38:02.223237 8939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-2/wal
uuid: "24f77571e2ef4fb39c11addd8d6d767e"
format_stamp: "Formatted at 2026-06-19 14:38:02 on dist-test-slave-32hj"
server_key: "fff3e3c796ea8c95d2aadcfc833ce91c"
server_key_iv: "8b517e9439b2a32a8fc776f87576733f"
server_key_version: "encryptionkey@0"
I20260619 14:38:02.223333 8939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:38:02.269767 8939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:38:02.273211 8939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:38:02.273392 8939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:38:02.273983 8939 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:38:02.274812 8939 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:38:02.274868 8939 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:38:02.274920 8939 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:38:02.274945 8939 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:38:02.282958 8939 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41737
I20260619 14:38:02.282979 9067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41737 every 8 connection(s)
I20260619 14:38:02.283813 8939 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:38:02.285346 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 8939
I20260619 14:38:02.285452 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:38:02.285751 392 external_mini_cluster.cc:1428] Setting key d5d9c9edbcc0a6bff880f6d6a916c336
Jun 19 14:38:02 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879882, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:38:02.297550 8648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:02.285717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43665 (local address 127.0.98.62:34179)
0619 14:38:02.285862 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:02.285867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:02.286816 (+ 949us) server_negotiation.cc:407] Connection header received
0619 14:38:02.288335 (+ 1519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:02.288338 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:02.288390 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:02.288465 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:02.289691 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:02.290205 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:02.291332 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:02.291614 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:02.291697 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:02.294294 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:38:02.294298 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:38:02.294301 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:38:02.294327 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:38:02.295590 (+ 1263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:38:02.296061 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:38:02.296065 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:38:02.296067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:38:02.296148 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:38:02.296447 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:38:02.296451 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:38:02.296453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:38:02.296658 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:38:02.296761 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 14:38:02.297269 (+ 508us) server_negotiation.cc:299] Negotiation successful
0619 14:38:02.297391 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 14:38:02.298357 9070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:02.286178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34179 (local address 127.0.98.3:43665)
0619 14:38:02.286669 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:02.286723 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:02.288154 (+ 1431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:02.288670 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:02.288677 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:02.289087 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:38:02.289562 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:02.289576 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:02.290312 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:02.290316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:02.291228 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:02.291237 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:02.291755 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:02.291759 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:02.291934 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:02.292500 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:38:02.292524 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:38:02.294135 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:38:02.295702 (+ 1567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:38:02.295708 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:38:02.295724 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:38:02.295971 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:38:02.296230 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:38:02.296233 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:38:02.296235 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:38:02.296367 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:38:02.296775 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:38:02.296781 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:38:02.297030 (+ 249us) client_negotiation.cc:769] Sending connection context
0619 14:38:02.297213 (+ 183us) client_negotiation.cc:240] Negotiation successful
0619 14:38:02.297533 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":128,"threads_started":1}
I20260619 14:38:02.299417 9068 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34179
I20260619 14:38:02.299670 9068 heartbeater.cc:464] Registering TS with master...
I20260619 14:38:02.300115 9068 heartbeater.cc:511] Master 127.0.98.62:34179 requested a full tablet report, sending...
I20260619 14:38:02.301059 8609 ts_manager.cc:194] Registered new tserver with Master: 24f77571e2ef4fb39c11addd8d6d767e (127.0.98.3:41737)
I20260619 14:38:02.301687 8609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43665
I20260619 14:38:02.311170 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260619 14:38:02.375036 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 8671
I20260619 14:38:02.379796 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 8805
I20260619 14:38:02.384348 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 8939
I20260619 14:38:02.388808 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 8577
I20260619 14:38:02.393653 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34179
--webserver_interface=127.0.98.62
--webserver_port=42997
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34179
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:38:02.499003 9077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:38:02.499269 9077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:38:02.499323 9077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:38:02.503301 9077 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:38:02.503371 9077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:38:02.503397 9077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:38:02.503425 9077 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:38:02.503446 9077 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:38:02.508694 9077 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34179
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=42997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:38:02.509797 9077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:38:02.510800 9077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:38:02.519207 9085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:38:02.519269 9077 server_base.cc:1034] running on GCE node
W20260619 14:38:02.519222 9082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:38:02.519219 9083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:38:02.519958 9077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:38:02.521082 9077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:38:02.522253 9077 hybrid_clock.cc:648] HybridClock initialized: now 1781879882522220 us; error 43 us; skew 500 ppm
Jun 19 14:38:02 dist-test-slave-32hj krb5kdc[8564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879882, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:38:02.525005 9077 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:38:02.525897 9077 webserver.cc:460] Webserver started at http://127.0.98.62:42997/ using document root <none> and password file <none>
I20260619 14:38:02.526429 9077 fs_manager.cc:359] Metadata directory not provided
I20260619 14:38:02.526507 9077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:38:02.530392 9077 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:38:02.532505 9092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:38:02.533838 9077 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:38:02.533939 9077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/master-0/wal
uuid: "f5e1fe3efbd848e6b60728f62f513fee"
format_stamp: "Formatted at 2026-06-19 14:38:01 on dist-test-slave-32hj"
server_key: "59578fd1aedd06b5dcaf5441efd0a577"
server_key_iv: "28e6002303b5d00de9e0e6cfc84d7f9c"
server_key_version: "encryptionkey@0"
I20260619 14:38:02.534487 9077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:38:02.555289 9077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:38:02.558332 9077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:38:02.558485 9077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:38:02.565302 9077 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34179
I20260619 14:38:02.565372 9144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34179 every 8 connection(s)
I20260619 14:38:02.566358 9077 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:38:02.569077 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 9077
I20260619 14:38:02.569464 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41541
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34179
--webserver_port=34195
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:38:02.570683 9145 sys_catalog.cc:263] Verifying existing consensus state
I20260619 14:38:02.572649 9145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5e1fe3efbd848e6b60728f62f513fee: Bootstrap starting.
I20260619 14:38:02.588680 9145 log.cc:824] T 00000000000000000000000000000000 P f5e1fe3efbd848e6b60728f62f513fee: Log is configured to *not* fsync() on all Append() calls
I20260619 14:38:02.595302 9145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5e1fe3efbd848e6b60728f62f513fee: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:38:02.595649 9145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5e1fe3efbd848e6b60728f62f513fee: Bootstrap complete.
I20260619 14:38:02.599588 9145 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5e1fe3efbd848e6b60728f62f513fee [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5e1fe3efbd848e6b60728f62f513fee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34179 } }
I20260619 14:38:02.599970 9145 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5e1fe3efbd848e6b60728f62f513fee [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5e1fe3efbd848e6b60728f62f513fee, State: Initialized, Role: FOLLOWER
I20260619 14:38:02.600425 9145 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5e1fe3efbd848e6b60728f62f513fee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5e1fe3efbd848e6b60728f62f513fee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34179 } }
I20260619 14:38:02.600540 9145 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5e1fe3efbd848e6b60728f62f513fee [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:38:02.600601 9145 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5e1fe3efbd848e6b60728f62f513fee [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:38:02.600688 9145 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5e1fe3efbd848e6b60728f62f513fee [term 1 FOLLOWER]: Advancing to term 2
I20260619 14:38:02.602507 9145 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5e1fe3efbd848e6b60728f62f513fee [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5e1fe3efbd848e6b60728f62f513fee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34179 } }
I20260619 14:38:02.602802 9145 leader_election.cc:302] T 00000000000000000000000000000000 P f5e1fe3efbd848e6b60728f62f513fee [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5e1fe3efbd848e6b60728f62f513fee; no voters:
I20260619 14:38:02.603113 9145 leader_election.cc:288] T 00000000000000000000000000000000 P f5e1fe3efbd848e6b60728f62f513fee [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 14:38:02.603302 9149 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5e1fe3efbd848e6b60728f62f513fee [term 2 FOLLOWER]: Leader election won for term 2
I20260619 14:38:02.603506 9149 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5e1fe3efbd848e6b60728f62f513fee [term 2 LEADER]: Becoming Leader. State: Replica: f5e1fe3efbd848e6b60728f62f513fee, State: Running, Role: LEADER
I20260619 14:38:02.603789 9149 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5e1fe3efbd848e6b60728f62f513fee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5e1fe3efbd848e6b60728f62f513fee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34179 } }
I20260619 14:38:02.604341 9145 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5e1fe3efbd848e6b60728f62f513fee [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:38:02.605098 9150 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5e1fe3efbd848e6b60728f62f513fee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f5e1fe3efbd848e6b60728f62f513fee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5e1fe3efbd848e6b60728f62f513fee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34179 } } }
I20260619 14:38:02.605221 9150 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5e1fe3efbd848e6b60728f62f513fee [sys.catalog]: This master's current role is: LEADER
I20260619 14:38:02.605516 9151 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5e1fe3efbd848e6b60728f62f513fee [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5e1fe3efbd848e6b60728f62f513fee. Latest consensus state: current_term: 2 leader_uuid: "f5e1fe3efbd848e6b60728f62f513fee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5e1fe3efbd848e6b60728f62f513fee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34179 } } }
I20260619 14:38:02.605652 9151 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5e1fe3efbd848e6b60728f62f513fee [sys.catalog]: This master's current role is: LEADER
I20260619 14:38:02.605974 9158 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:38:02.609102 9158 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:38:02.609925 9158 catalog_manager.cc:1259] Loaded cluster ID: 2a063f592c5045b49b7ffbb3ff82472f
I20260619 14:38:02.609979 9158 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:38:02.612265 9158 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:38:02.612982 9158 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f5e1fe3efbd848e6b60728f62f513fee: Loaded TSK: 0
I20260619 14:38:02.614274 9158 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 14:38:02.680321 9147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:38:02.680588 9147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:38:02.680627 9147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:38:02.684614 9147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:38:02.684680 9147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:38:02.684765 9147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:38:02.689975 9147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41541
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=34195
--tserver_master_addrs=127.0.98.62:34179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:38:02.690991 9147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:38:02.692003 9147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:38:02.702188 9171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:38:02.702188 9172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:38:02.702212 9174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:38:02.702461 9147 server_base.cc:1034] running on GCE node
I20260619 14:38:02.703008 9147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:38:02.703780 9147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:38:02.704972 9147 hybrid_clock.cc:648] HybridClock initialized: now 1781879882704957 us; error 32 us; skew 500 ppm
Jun 19 14:38:02 dist-test-slave-32hj krb5kdc[8564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879882, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:38:02.708359 9147 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260619 14:38:02.709486 9147 webserver.cc:460] Webserver started at http://127.0.98.1:34195/ using document root <none> and password file <none>
I20260619 14:38:02.710217 9147 fs_manager.cc:359] Metadata directory not provided
I20260619 14:38:02.710325 9147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:38:02.715349 9147 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:38:02.717918 9181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:38:02.719386 9147 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:38:02.719508 9147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-0/wal
uuid: "6f8a85683eb74926b43ae1d5ef6337f1"
format_stamp: "Formatted at 2026-06-19 14:38:01 on dist-test-slave-32hj"
server_key: "afb2b09909732ac09d9a65058c3b5a68"
server_key_iv: "f366239fc8d873f56a3a2b6781c4e969"
server_key_version: "encryptionkey@0"
I20260619 14:38:02.720224 9147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:38:02.737548 9147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:38:02.740586 9147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:38:02.740783 9147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:38:02.741392 9147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:38:02.742221 9147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:38:02.742266 9147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:38:02.742303 9147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:38:02.742326 9147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:38:02.750952 9147 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41541
I20260619 14:38:02.750986 9294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41541 every 8 connection(s)
I20260619 14:38:02.751924 9147 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:38:02.755571 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 9147
I20260619 14:38:02.755992 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:35537
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34179
--webserver_port=46673
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:38:02 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:38:02 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 14:38:02.765331 9297 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:38:02.753982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35757 (local address 127.0.98.62:34179)
0619 14:38:02.754479 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:02.754498 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:02.754717 (+ 219us) server_negotiation.cc:407] Connection header received
0619 14:38:02.756265 (+ 1548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:02.756280 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:02.756674 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:02.757055 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:02.758422 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:02.759384 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:02.760616 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:02.761125 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:02.761275 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:02.764588 (+ 3313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35757: BlockingRecv error: recv got EOF from 127.0.98.1:35757 (error 108)
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":162,"threads_started":1}
I20260619 14:38:02.765338 9298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:02.754061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34179 (local address 127.0.98.1:35757)
0619 14:38:02.754548 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:02.754599 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:02.755826 (+ 1227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:02.757123 (+ 1297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:02.757129 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:02.757586 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:38:02.758257 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:02.758277 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:02.759502 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:02.759504 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:02.760494 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:02.760504 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:02.761274 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:02.761278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:02.761454 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:02.762026 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:38:02.762042 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:38:02.764364 (+ 2322us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":136,"threads_started":1}
W20260619 14:38:02.766037 9295 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34179 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:34179: Client connection negotiation failed: client connection to 127.0.98.62:34179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 19 14:38:02 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:38:02 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:38:02.771136 9298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:02.766484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34179 (local address 127.0.98.1:37317)
0619 14:38:02.766582 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:02.766595 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:02.766664 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:02.766864 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:02.766866 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:02.767025 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:38:02.767105 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:02.767110 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:02.767780 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:02.767783 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:02.768310 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:02.768317 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:02.768709 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:02.768713 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:02.768816 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:02.769299 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:38:02.769311 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:38:02.771009 (+ 1698us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260619 14:38:02.771234 9297 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:38:02.766558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37317 (local address 127.0.98.62:34179)
0619 14:38:02.766664 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:02.766668 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:02.766681 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:38:02.766739 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:02.766742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:02.766781 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:02.766842 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:02.767204 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:02.767687 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:02.768408 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:02.768619 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:02.768702 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:02.771139 (+ 2437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37317: BlockingRecv error: recv got EOF from 127.0.98.1:37317 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Jun 19 14:38:02 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:38:02 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:38:02.776790 9298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:02.771900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34179 (local address 127.0.98.1:50897)
0619 14:38:02.772030 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:02.772044 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:02.772114 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:02.772320 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:02.772323 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:02.772472 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:38:02.772555 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:02.772560 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:02.773227 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:02.773230 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:02.774100 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:02.774109 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:02.774473 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:02.774476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:02.774559 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:02.774960 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:38:02.774973 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:38:02.776627 (+ 1654us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":59}
W20260619 14:38:02.776851 9297 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:38:02.772002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50897 (local address 127.0.98.62:34179)
0619 14:38:02.772095 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:02.772098 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:02.772111 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:38:02.772198 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:02.772200 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:02.772234 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:02.772293 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:02.772643 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:02.773143 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:02.774198 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:02.774390 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:02.774453 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:02.776760 (+ 2307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50897: BlockingRecv error: recv got EOF from 127.0.98.1:50897 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260619 14:38:02.776988 9295 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 14:38:02.866564 9299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:38:02.866827 9299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:38:02.866868 9299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:38:02.870864 9299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:38:02.870923 9299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:38:02.870982 9299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:38:02.876243 9299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:35537
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=46673
--tserver_master_addrs=127.0.98.62:34179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:38:02.877295 9299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:38:02.878283 9299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:38:02.887470 9307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:38:02.887480 9305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:38:02.887462 9304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:38:02.887725 9299 server_base.cc:1034] running on GCE node
I20260619 14:38:02.888125 9299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:38:02.888742 9299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:38:02.889907 9299 hybrid_clock.cc:648] HybridClock initialized: now 1781879882889877 us; error 43 us; skew 500 ppm
Jun 19 14:38:02 dist-test-slave-32hj krb5kdc[8564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879882, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:38:02.892508 9299 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260619 14:38:02.893361 9299 webserver.cc:460] Webserver started at http://127.0.98.2:46673/ using document root <none> and password file <none>
I20260619 14:38:02.893859 9299 fs_manager.cc:359] Metadata directory not provided
I20260619 14:38:02.893937 9299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:38:02.897850 9299 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:38:02.899850 9314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:38:02.901155 9299 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:38:02.901268 9299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-1/wal
uuid: "722677b14e094f2d93709caa5aac50b5"
format_stamp: "Formatted at 2026-06-19 14:38:01 on dist-test-slave-32hj"
server_key: "27fe22de353df4994616cf1275755ef1"
server_key_iv: "048b6f2ba3252320a1a9edac5b8ba18c"
server_key_version: "encryptionkey@0"
I20260619 14:38:02.901819 9299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:38:02.914984 9299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:38:02.918094 9299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:38:02.918256 9299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:38:02.918823 9299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:38:02.919836 9299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:38:02.919884 9299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:38:02.919936 9299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:38:02.919965 9299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:38:02.928395 9299 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35537
I20260619 14:38:02.928431 9427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35537 every 8 connection(s)
I20260619 14:38:02.929430 9299 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:38:02.933293 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 9299
I20260619 14:38:02.933760 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:41737
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34179
--webserver_port=45249
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:38:02 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:38:02 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:38:02.942022 9430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:02.931544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34179 (local address 127.0.98.2:40471)
0619 14:38:02.931977 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:02.932019 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:02.933510 (+ 1491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:02.934179 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:02.934188 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:02.934650 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:38:02.935289 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:02.935305 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:02.936112 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:02.936116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:02.937051 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:02.937060 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:02.937567 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:02.937571 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:02.937735 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:02.938321 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:38:02.938337 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:38:02.940895 (+ 2558us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":110,"threads_started":1}
W20260619 14:38:02.942268 9297 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:38:02.931171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40471 (local address 127.0.98.62:34179)
0619 14:38:02.931313 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:02.931316 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:02.932126 (+ 810us) server_negotiation.cc:407] Connection header received
0619 14:38:02.933779 (+ 1653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:02.933784 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:02.933848 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:02.933943 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:02.935429 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:02.936003 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:02.937165 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:02.937420 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:02.937524 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:02.942158 (+ 4634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:40471: BlockingRecv error: recv got EOF from 127.0.98.2:40471 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260619 14:38:02.942695 9428 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34179 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:34179: Client connection negotiation failed: client connection to 127.0.98.62:34179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 19 14:38:02 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:38:02 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:38:02.947923 9430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:02.943155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34179 (local address 127.0.98.2:58717)
0619 14:38:02.943272 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:02.943286 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:02.943367 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:02.943587 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:02.943590 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:02.943740 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:38:02.943825 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:02.943830 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:02.944511 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:02.944515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:02.945115 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:02.945124 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:02.945518 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:02.945521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:02.945604 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:02.946070 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:38:02.946081 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:38:02.947830 (+ 1749us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 14:38:02.948204 9297 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:38:02.943205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58717 (local address 127.0.98.62:34179)
0619 14:38:02.943309 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:02.943312 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:02.943326 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:38:02.943458 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:02.943461 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:02.943501 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:02.943571 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:02.943928 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:02.944414 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:02.945218 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:02.945426 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:02.945511 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:02.947974 (+ 2463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58717: BlockingRecv error: recv got EOF from 127.0.98.2:58717 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 19 14:38:02 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:38:02 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:38:02.953226 9430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:02.948681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34179 (local address 127.0.98.2:55101)
0619 14:38:02.948800 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:02.948813 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:02.948883 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:02.949104 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:02.949107 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:02.949237 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:38:02.949300 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:02.949304 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:02.950000 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:02.950003 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:02.950515 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:02.950522 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:02.950917 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:02.950920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:02.951000 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:02.951430 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:38:02.951440 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:38:02.953115 (+ 1675us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
W20260619 14:38:02.953351 9297 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:38:02.948797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55101 (local address 127.0.98.62:34179)
0619 14:38:02.948909 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:02.948913 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:02.948937 (+ 24us) server_negotiation.cc:407] Connection header received
0619 14:38:02.948981 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:02.948984 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:02.949018 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:02.949077 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:02.949395 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:02.949888 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:02.950612 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:02.950825 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:02.950907 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:02.953255 (+ 2348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55101: BlockingRecv error: recv got EOF from 127.0.98.2:55101 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260619 14:38:02.953583 9428 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 14:38:03.046494 9431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:38:03.046778 9431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:38:03.046823 9431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:38:03.051055 9431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:38:03.051119 9431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:38:03.051180 9431 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:38:03.056463 9431 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:41737
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=45249
--tserver_master_addrs=127.0.98.62:34179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9431
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:38:03.057569 9431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:38:03.058525 9431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:38:03.067750 9439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:38:03.067764 9436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:38:03.067801 9437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:38:03.067930 9431 server_base.cc:1034] running on GCE node
I20260619 14:38:03.068405 9431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:38:03.069032 9431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:38:03.070204 9431 hybrid_clock.cc:648] HybridClock initialized: now 1781879883070171 us; error 42 us; skew 500 ppm
Jun 19 14:38:03 dist-test-slave-32hj krb5kdc[8564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879883, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:38:03.073078 9431 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260619 14:38:03.073921 9431 webserver.cc:460] Webserver started at http://127.0.98.3:45249/ using document root <none> and password file <none>
I20260619 14:38:03.074432 9431 fs_manager.cc:359] Metadata directory not provided
I20260619 14:38:03.074502 9431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:38:03.078269 9431 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:38:03.080307 9446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:38:03.081596 9431 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:38:03.081691 9431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-2/wal
uuid: "24f77571e2ef4fb39c11addd8d6d767e"
format_stamp: "Formatted at 2026-06-19 14:38:02 on dist-test-slave-32hj"
server_key: "fff3e3c796ea8c95d2aadcfc833ce91c"
server_key_iv: "8b517e9439b2a32a8fc776f87576733f"
server_key_version: "encryptionkey@0"
I20260619 14:38:03.082221 9431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:38:03.101397 9431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:38:03.104566 9431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:38:03.104771 9431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:38:03.105399 9431 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:38:03.106310 9431 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:38:03.106356 9431 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:38:03.106400 9431 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:38:03.106432 9431 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:38:03.115332 9431 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41737
I20260619 14:38:03.115355 9559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41737 every 8 connection(s)
I20260619 14:38:03.116340 9431 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:38:03.120358 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 9431
Jun 19 14:38:03 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:38:03 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 14:38:03.128094 9297 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:38:03.118127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44563 (local address 127.0.98.62:34179)
0619 14:38:03.118241 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:03.118244 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:03.119207 (+ 963us) server_negotiation.cc:407] Connection header received
0619 14:38:03.120665 (+ 1458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:03.120668 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:03.120744 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:03.120829 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:03.122324 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:03.122912 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:03.124166 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:03.124425 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:03.124524 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:03.127954 (+ 3430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:44563: BlockingRecv error: recv got EOF from 127.0.98.3:44563 (error 108)
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":39}
I20260619 14:38:03.128825 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:03.118494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34179 (local address 127.0.98.3:44563)
0619 14:38:03.119014 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:03.119065 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:03.120428 (+ 1363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:03.121050 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:03.121059 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:03.121566 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:38:03.122184 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:03.122199 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:03.123027 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:03.123031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:03.124042 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:03.124052 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:03.124571 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:03.124574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:03.124746 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:03.125243 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:38:03.125260 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:38:03.127841 (+ 2581us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":142,"threads_started":1}
W20260619 14:38:03.129251 9560 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34179 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:34179: Client connection negotiation failed: client connection to 127.0.98.62:34179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260619 14:38:03.130285 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:03.121335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38640 (local address 127.0.98.62:34179)
0619 14:38:03.121585 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:03.121589 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:03.121636 (+ 47us) server_negotiation.cc:407] Connection header received
0619 14:38:03.121737 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:03.121740 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:03.121782 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:03.121871 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:03.122574 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:03.123137 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:03.124038 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:03.124352 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:03.124425 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:03.125875 (+ 1450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:38:03.125881 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:38:03.125895 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:38:03.125921 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:38:03.127547 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:38:03.127957 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:38:03.127963 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:38:03.127968 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:38:03.128095 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:38:03.128457 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:38:03.128461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:38:03.128463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:38:03.128833 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:38:03.129036 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0619 14:38:03.129757 (+ 721us) server_negotiation.cc:299] Negotiation successful
0619 14:38:03.130140 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":167,"thread_start_us":70,"threads_started":1}
Jun 19 14:38:03 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:38:03 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:38:03.135053 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:03.129676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34179 (local address 127.0.98.3:38347)
0619 14:38:03.129843 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:03.129856 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:03.129937 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:03.130194 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:03.130197 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:03.130366 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:38:03.130510 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:03.130518 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:03.131415 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:03.131418 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:03.132227 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:03.132238 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:03.132618 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:03.132621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:03.132710 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:03.133290 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:38:03.133305 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:38:03.134956 (+ 1651us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106}
W20260619 14:38:03.135190 9297 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:38:03.129748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38347 (local address 127.0.98.62:34179)
0619 14:38:03.129856 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:03.129860 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:03.129909 (+ 49us) server_negotiation.cc:407] Connection header received
0619 14:38:03.130009 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:03.130012 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:03.130055 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:03.130132 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:03.130621 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:03.131304 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:03.132323 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:03.132526 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:03.132616 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:03.135085 (+ 2469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38347: BlockingRecv error: recv got EOF from 127.0.98.3:38347 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 19 14:38:03 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:38:03 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:38:03.140260 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:03.135540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34179 (local address 127.0.98.3:58711)
0619 14:38:03.135677 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:03.135691 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:03.135759 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:03.135938 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:03.135940 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:03.136055 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:38:03.136128 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:03.136132 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:03.136893 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:03.136896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:03.137445 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:03.137452 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:03.137947 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:03.137950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:03.138014 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:03.138453 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:38:03.138464 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:38:03.140156 (+ 1692us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 14:38:03.140394 9297 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:38:03.135624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58711 (local address 127.0.98.62:34179)
0619 14:38:03.135758 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:03.135761 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:03.135773 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:38:03.135822 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:03.135824 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:03.135857 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:03.135912 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:03.136222 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:03.136786 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:03.137571 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:03.137826 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:03.137913 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:03.140287 (+ 2374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58711: BlockingRecv error: recv got EOF from 127.0.98.3:58711 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260619 14:38:03.140507 9560 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 19 14:38:03 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:38:03 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:38:03.782405 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:03.777497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34179 (local address 127.0.98.1:33679)
0619 14:38:03.777779 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:03.777794 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:03.777881 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:03.778080 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:03.778082 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:03.778246 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:38:03.778358 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:03.778363 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:03.779020 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:03.779022 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:03.779563 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:03.779569 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:03.779978 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:03.779981 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:03.780069 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:03.780534 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:38:03.780546 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:38:03.782286 (+ 1740us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":78,"threads_started":1}
W20260619 14:38:03.782513 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:38:03.777580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33679 (local address 127.0.98.62:34179)
0619 14:38:03.777828 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:03.777831 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:03.777843 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:38:03.777946 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:03.777948 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:03.777989 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:03.778058 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:03.778440 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:03.778946 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:03.779652 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:03.779861 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:03.779932 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:03.782406 (+ 2474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33679: BlockingRecv error: recv got EOF from 127.0.98.1:33679 (error 108)
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":105,"threads_started":1}
Jun 19 14:38:03 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:38:03 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:38:03.959491 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:03.954316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34179 (local address 127.0.98.2:35845)
0619 14:38:03.954615 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:03.954628 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:03.954748 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:03.954992 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:03.954995 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:03.955194 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:38:03.955297 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:03.955303 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:03.956054 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:03.956058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:03.956640 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:03.956649 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:03.957105 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:03.957108 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:03.957198 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:03.957686 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:38:03.957698 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:38:03.959336 (+ 1638us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":79,"threads_started":1}
W20260619 14:38:03.959565 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:38:03.954395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35845 (local address 127.0.98.62:34179)
0619 14:38:03.954529 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:03.954533 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:03.954713 (+ 180us) server_negotiation.cc:407] Connection header received
0619 14:38:03.954837 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:03.954840 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:03.954888 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:03.954954 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:03.955414 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:03.955950 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:03.956770 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:03.957001 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:03.957088 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:03.959466 (+ 2378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35845: BlockingRecv error: recv got EOF from 127.0.98.2:35845 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 19 14:38:04 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:38:04 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:38:04.146296 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:04.141128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34179 (local address 127.0.98.3:55471)
0619 14:38:04.141375 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:04.141388 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:04.141482 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:04.141744 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:04.141747 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:04.141932 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:38:04.142050 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:04.142062 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:04.142848 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:04.142851 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:04.143452 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:04.143461 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:04.143885 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:04.143887 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:04.143966 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:04.144433 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:38:04.144446 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:38:04.146171 (+ 1725us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":73,"threads_started":1}
W20260619 14:38:04.146400 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:38:04.141184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55471 (local address 127.0.98.62:34179)
0619 14:38:04.141296 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:04.141300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:04.141454 (+ 154us) server_negotiation.cc:407] Connection header received
0619 14:38:04.141551 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:04.141553 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:04.141607 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:04.141674 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:04.142168 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:04.142731 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:04.143560 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:04.143799 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:04.143878 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:04.146300 (+ 2422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55471: BlockingRecv error: recv got EOF from 127.0.98.3:55471 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Jun 19 14:38:04 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:38:04 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:38:04.788556 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:04.783178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34179 (local address 127.0.98.1:43643)
0619 14:38:04.783451 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:04.783466 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:04.783557 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:04.783838 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:04.783841 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:04.784037 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:38:04.784151 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:04.784158 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:04.784958 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:04.784961 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:04.785547 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:04.785557 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:04.786068 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:04.786072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:04.786156 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:04.786695 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:38:04.786710 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:38:04.788424 (+ 1714us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":72,"threads_started":1}
W20260619 14:38:04.788699 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:38:04.783267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43643 (local address 127.0.98.62:34179)
0619 14:38:04.783502 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:04.783505 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:04.783520 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:38:04.783638 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:04.783640 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:04.783682 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:04.783763 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:04.784257 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:04.784844 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:04.785676 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:04.785952 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:04.786054 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:04.788548 (+ 2494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43643: BlockingRecv error: recv got EOF from 127.0.98.1:43643 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":67,"threads_started":1}
Jun 19 14:38:04 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:38:04 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:38:04.965415 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:04.960242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34179 (local address 127.0.98.2:36873)
0619 14:38:04.960507 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:04.960524 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:04.960639 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:04.960887 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:04.960889 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:04.961082 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:38:04.961173 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:04.961179 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:04.961902 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:04.961905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:04.962485 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:04.962494 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:04.962916 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:04.962919 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:04.963006 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:04.963497 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:38:04.963516 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:38:04.965213 (+ 1697us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":82,"threads_started":1}
W20260619 14:38:04.965451 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:38:04.960323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36873 (local address 127.0.98.62:34179)
0619 14:38:04.960511 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:04.960516 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:04.960596 (+ 80us) server_negotiation.cc:407] Connection header received
0619 14:38:04.960738 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:04.960742 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:04.960790 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:04.960856 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:04.961285 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:04.961804 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:04.962599 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:04.962822 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:04.962905 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:04.965351 (+ 2446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36873: BlockingRecv error: recv got EOF from 127.0.98.2:36873 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Jun 19 14:38:05 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:38:05 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:38:05.152036 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:05.147017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34179 (local address 127.0.98.3:46919)
0619 14:38:05.147330 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:05.147344 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:05.147445 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:05.147703 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:05.147706 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:05.147877 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:38:05.147985 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:05.147991 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:05.148678 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:05.148681 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:05.149276 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:05.149283 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:05.149658 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:05.149661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:05.149720 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:05.150213 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:38:05.150225 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:38:05.151910 (+ 1685us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":113,"threads_started":1}
W20260619 14:38:05.152168 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:38:05.147196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46919 (local address 127.0.98.62:34179)
0619 14:38:05.147330 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:05.147334 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:05.147395 (+ 61us) server_negotiation.cc:407] Connection header received
0619 14:38:05.147521 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:05.147523 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:05.147572 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:05.147630 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:05.148078 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:05.148589 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:05.149362 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:05.149575 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:05.149647 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:05.152048 (+ 2401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46919: BlockingRecv error: recv got EOF from 127.0.98.3:46919 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 19 14:38:05 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:38:05 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:38:05.794013 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:05.789201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34179 (local address 127.0.98.1:46155)
0619 14:38:05.789478 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:05.789492 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:05.789598 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:05.789805 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:05.789807 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:05.789981 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:38:05.790075 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:05.790080 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:05.790739 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:05.790741 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:05.791293 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:05.791299 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:05.791648 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:05.791650 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:05.791705 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:05.792155 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:38:05.792166 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:38:05.793847 (+ 1681us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":82,"threads_started":1}
W20260619 14:38:05.794111 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:38:05.789295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46155 (local address 127.0.98.62:34179)
0619 14:38:05.789501 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:05.789504 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:05.789563 (+ 59us) server_negotiation.cc:407] Connection header received
0619 14:38:05.789665 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:05.789667 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:05.789709 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:05.789780 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:05.790156 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:05.790665 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:05.791369 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:05.791574 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:05.791638 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:05.793984 (+ 2346us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46155: BlockingRecv error: recv got EOF from 127.0.98.1:46155 (error 108)
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":51,"threads_started":1}
Jun 19 14:38:05 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:38:05 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:38:05.970929 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:05.966134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34179 (local address 127.0.98.2:36465)
0619 14:38:05.966377 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:05.966390 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:05.966486 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:05.966692 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:05.966694 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:05.966852 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:38:05.966944 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:05.966949 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:05.967653 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:05.967656 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:05.968223 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:05.968232 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:05.968589 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:05.968591 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:05.968646 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:05.969128 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:38:05.969140 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:38:05.970816 (+ 1676us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":84,"threads_started":1}
W20260619 14:38:05.971076 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:38:05.966181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36465 (local address 127.0.98.62:34179)
0619 14:38:05.966295 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:05.966298 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:05.966484 (+ 186us) server_negotiation.cc:407] Connection header received
0619 14:38:05.966552 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:05.966555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:05.966602 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:05.966669 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:05.967040 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:05.967559 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:05.968308 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:05.968509 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:05.968574 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:05.970955 (+ 2381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36465: BlockingRecv error: recv got EOF from 127.0.98.2:36465 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Jun 19 14:38:06 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:38:06 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:38:06.157891 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:06.152794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34179 (local address 127.0.98.3:48525)
0619 14:38:06.153058 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:06.153071 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:06.153166 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:06.153372 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:06.153374 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:06.153541 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:38:06.153636 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:06.153641 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:06.154312 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:06.154315 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:06.154881 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:06.154889 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:06.155311 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:06.155315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:06.155386 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:06.155835 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:38:06.155848 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:38:06.157720 (+ 1872us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":91,"threads_started":1}
W20260619 14:38:06.157956 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:38:06.152879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48525 (local address 127.0.98.62:34179)
0619 14:38:06.153005 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:06.153009 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:06.153132 (+ 123us) server_negotiation.cc:407] Connection header received
0619 14:38:06.153231 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:06.153233 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:06.153279 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:06.153342 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:06.153713 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:06.154219 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:06.154981 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:06.155194 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:06.155270 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:06.157857 (+ 2587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:48525: BlockingRecv error: recv got EOF from 127.0.98.3:48525 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Jun 19 14:38:06 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:38:06 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:38:06.804711 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:06.794800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34179 (local address 127.0.98.1:41685)
0619 14:38:06.795114 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:06.795128 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:06.795250 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:06.795580 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:06.795584 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:06.795787 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:38:06.795905 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:06.795914 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:06.796868 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:06.796871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:06.797420 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:06.797428 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:06.797844 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:06.797846 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:06.797904 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:06.802675 (+ 4771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:38:06.802691 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:38:06.804572 (+ 1881us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":98,"threads_started":1}
W20260619 14:38:06.805015 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:38:06.794917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41685 (local address 127.0.98.62:34179)
0619 14:38:06.795168 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:06.795172 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:06.795195 (+ 23us) server_negotiation.cc:407] Connection header received
0619 14:38:06.795344 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:06.795346 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:06.795396 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:06.795492 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:06.796040 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:06.796777 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:06.797533 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:06.797758 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:06.797836 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:06.804838 (+ 7002us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41685: BlockingRecv error: recv got EOF from 127.0.98.1:41685 (error 108)
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":83,"threads_started":1}
Jun 19 14:38:06 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:38:06 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:38:06.976540 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:06.971661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34179 (local address 127.0.98.2:34503)
0619 14:38:06.971936 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:06.971948 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:06.972045 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:06.972306 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:06.972308 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:06.972475 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:38:06.972572 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:06.972577 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:06.973283 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:06.973286 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:06.973853 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:06.973860 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:06.974226 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:06.974229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:06.974285 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:06.974770 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:38:06.974781 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:38:06.976424 (+ 1643us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":88,"threads_started":1}
W20260619 14:38:06.976682 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:38:06.971730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34503 (local address 127.0.98.62:34179)
0619 14:38:06.971851 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:06.971855 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:06.972011 (+ 156us) server_negotiation.cc:407] Connection header received
0619 14:38:06.972127 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:06.972129 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:06.972171 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:06.972238 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:06.972662 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:06.973202 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:06.973936 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:06.974150 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:06.974219 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:06.976564 (+ 2345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34503: BlockingRecv error: recv got EOF from 127.0.98.2:34503 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Jun 19 14:38:07 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:38:07 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:38:07.163887 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:07.158754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34179 (local address 127.0.98.3:41587)
0619 14:38:07.159009 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:07.159026 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:07.159136 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:07.159366 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:07.159369 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:07.159588 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:38:07.159679 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:07.159685 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:07.160409 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:07.160412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:07.161016 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:07.161024 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:07.161443 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:07.161447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:07.161535 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:07.161984 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:38:07.161995 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:38:07.163731 (+ 1736us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":82,"threads_started":1}
W20260619 14:38:07.163973 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:38:07.158858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41587 (local address 127.0.98.62:34179)
0619 14:38:07.159010 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:07.159015 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:07.159081 (+ 66us) server_negotiation.cc:407] Connection header received
0619 14:38:07.159219 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:07.159222 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:07.159270 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:07.159335 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:07.159785 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:07.160311 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:07.161121 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:07.161347 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:07.161432 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:07.163858 (+ 2426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41587: BlockingRecv error: recv got EOF from 127.0.98.3:41587 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 19 14:38:07 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:38:07 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:38:07.810708 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:07.805505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34179 (local address 127.0.98.1:56265)
0619 14:38:07.805818 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:07.805832 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:07.805925 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:07.806210 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:07.806212 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:07.806385 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:38:07.806498 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:07.806503 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:07.807234 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:07.807236 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:07.807800 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:07.807805 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:07.808169 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:07.808171 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:07.808229 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:07.808674 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:38:07.808685 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:38:07.810559 (+ 1874us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":93,"threads_started":1}
W20260619 14:38:07.810810 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:38:07.805585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56265 (local address 127.0.98.62:34179)
0619 14:38:07.805831 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:07.805835 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:07.805893 (+ 58us) server_negotiation.cc:407] Connection header received
0619 14:38:07.806051 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:07.806055 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:07.806110 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:07.806190 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:07.806622 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:07.807156 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:07.807879 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:07.808091 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:07.808159 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:07.810694 (+ 2535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56265: BlockingRecv error: recv got EOF from 127.0.98.1:56265 (error 108)
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":69,"threads_started":1}
Jun 19 14:38:07 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:38:07 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:38:07.982497 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:07.977296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34179 (local address 127.0.98.2:34753)
0619 14:38:07.977581 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:07.977597 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:07.977715 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:07.977969 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:07.977971 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:07.978156 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:38:07.978246 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:07.978252 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:07.978994 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:07.978997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:07.979556 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:07.979564 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:07.979975 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:07.979978 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:07.980052 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:07.980548 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:38:07.980559 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:38:07.982312 (+ 1753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":78,"threads_started":1}
W20260619 14:38:07.982583 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:38:07.977453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34753 (local address 127.0.98.62:34179)
0619 14:38:07.977605 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:07.977609 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:07.977675 (+ 66us) server_negotiation.cc:407] Connection header received
0619 14:38:07.977802 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:07.977805 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:07.977860 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:07.977932 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:07.978355 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:07.978893 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:07.979664 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:07.979881 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:07.979966 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:07.982478 (+ 2512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34753: BlockingRecv error: recv got EOF from 127.0.98.2:34753 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 19 14:38:08 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:38:08 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:38:08.169915 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:08.164742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34179 (local address 127.0.98.3:54067)
0619 14:38:08.164995 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:08.165010 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:08.165108 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:08.165312 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:08.165314 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:08.165481 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:38:08.165600 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:08.165605 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:08.166323 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:08.166324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:08.166883 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:08.166889 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:08.167245 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:08.167247 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:08.167303 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:08.167820 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:38:08.167834 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:38:08.169764 (+ 1930us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":91,"threads_started":1}
W20260619 14:38:08.169996 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:38:08.164809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54067 (local address 127.0.98.62:34179)
0619 14:38:08.164946 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:08.164951 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:08.165078 (+ 127us) server_negotiation.cc:407] Connection header received
0619 14:38:08.165175 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:08.165178 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:08.165224 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:08.165289 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:08.165720 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:08.166243 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:08.166963 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:08.167167 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:08.167236 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:08.169897 (+ 2661us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54067: BlockingRecv error: recv got EOF from 127.0.98.3:54067 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 19 14:38:08 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:38:08 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:38:08.816355 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:08.811499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34179 (local address 127.0.98.1:38905)
0619 14:38:08.811737 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:08.811750 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:08.811836 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:08.812057 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:08.812059 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:08.812225 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:38:08.812319 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:08.812324 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:08.813024 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:08.813026 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:08.813579 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:08.813587 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:08.813991 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:08.813993 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:08.814050 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:08.814562 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:38:08.814574 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:38:08.816223 (+ 1649us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":69,"threads_started":1}
W20260619 14:38:08.816591 9586 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:38:08.811570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38905 (local address 127.0.98.62:34179)
0619 14:38:08.811787 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:08.811790 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:08.811802 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:38:08.811908 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:08.811910 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:08.811952 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:08.812036 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:08.812414 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:08.812948 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:08.813676 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:08.813911 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:08.813984 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:08.816440 (+ 2456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38905: BlockingRecv error: recv got EOF from 127.0.98.1:38905 (error 108)
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":65,"threads_started":1}
Jun 19 14:38:08 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:38:08 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:38:08.988106 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:08.983265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34179 (local address 127.0.98.2:57961)
0619 14:38:08.983504 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:08.983518 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:08.983611 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:08.983830 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:08.983832 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:08.984026 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:38:08.984124 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:08.984129 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:08.984829 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:08.984831 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:08.985421 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:08.985429 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:08.985857 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:08.985860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:08.985916 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:08.986391 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:38:08.986402 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:38:08.987987 (+ 1585us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":71,"threads_started":1}
W20260619 14:38:08.988240 9586 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:38:08.983319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57961 (local address 127.0.98.62:34179)
0619 14:38:08.983446 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:08.983449 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:08.983574 (+ 125us) server_negotiation.cc:407] Connection header received
0619 14:38:08.983695 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:08.983697 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:08.983739 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:08.983797 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:08.984208 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:08.984754 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:08.985532 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:08.985772 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:08.985850 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:08.988124 (+ 2274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57961: BlockingRecv error: recv got EOF from 127.0.98.2:57961 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 19 14:38:09 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:38:09 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:38:09.175644 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:09.170719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34179 (local address 127.0.98.3:44499)
0619 14:38:09.170972 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:09.170987 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:09.171080 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:09.171279 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:09.171281 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:09.171467 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:38:09.171562 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:09.171567 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:09.172250 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:09.172252 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:09.172857 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:09.172866 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:09.173252 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:09.173254 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:09.173308 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:09.173769 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:38:09.173781 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:38:09.175469 (+ 1688us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":79,"threads_started":1}
W20260619 14:38:09.175714 9586 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:38:09.170781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44499 (local address 127.0.98.62:34179)
0619 14:38:09.170894 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:09.170898 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:09.171046 (+ 148us) server_negotiation.cc:407] Connection header received
0619 14:38:09.171144 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:09.171146 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:09.171191 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:09.171255 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:09.171641 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:09.172157 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:09.172949 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:09.173169 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:09.173246 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:09.175616 (+ 2370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:44499: BlockingRecv error: recv got EOF from 127.0.98.3:44499 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Jun 19 14:38:09 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:38:09 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:38:09.822561 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:09.817147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34179 (local address 127.0.98.1:55141)
0619 14:38:09.817423 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:09.817435 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:09.817521 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:09.817784 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:09.817787 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:09.817993 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:38:09.818099 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:09.818106 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:09.818877 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:09.818880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:09.819429 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:09.819438 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:09.819946 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:09.819949 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:09.820017 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:09.820523 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:38:09.820537 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:38:09.822413 (+ 1876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":66,"threads_started":1}
W20260619 14:38:09.822721 9590 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:38:09.817235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55141 (local address 127.0.98.62:34179)
0619 14:38:09.817475 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:09.817479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:09.817493 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:38:09.817589 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:09.817591 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:09.817632 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:09.817718 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:09.818218 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:09.818754 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:09.819558 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:09.819791 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:09.819899 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:09.822578 (+ 2679us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55141: BlockingRecv error: recv got EOF from 127.0.98.1:55141 (error 108)
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":91,"threads_started":1}
Jun 19 14:38:09 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:38:09 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:38:09.993661 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:09.988808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34179 (local address 127.0.98.2:45485)
0619 14:38:09.989074 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:09.989087 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:09.989182 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:09.989384 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:09.989386 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:09.989552 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:38:09.989645 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:09.989650 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:09.990314 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:09.990316 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:09.990874 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:09.990882 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:09.991231 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:09.991233 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:09.991286 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:09.991792 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:38:09.991805 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:38:09.993511 (+ 1706us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":69,"threads_started":1}
W20260619 14:38:09.993745 9590 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:38:09.988869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45485 (local address 127.0.98.62:34179)
0619 14:38:09.989006 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:09.989009 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:09.989145 (+ 136us) server_negotiation.cc:407] Connection header received
0619 14:38:09.989249 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:09.989251 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:09.989291 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:09.989355 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:09.989725 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:09.990225 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:09.990957 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:09.991159 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:09.991225 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:09.993634 (+ 2409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45485: BlockingRecv error: recv got EOF from 127.0.98.2:45485 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 19 14:38:10 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:38:10 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:38:10.181908 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:10.176549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34179 (local address 127.0.98.3:58235)
0619 14:38:10.176867 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:10.176881 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:10.176996 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:10.177212 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:10.177214 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:10.177379 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:38:10.177468 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:10.177472 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:10.178207 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:10.178210 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:10.178767 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:10.178774 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:10.179188 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:10.179191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:10.179273 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:10.179876 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:38:10.179888 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:38:10.181749 (+ 1861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":71,"threads_started":1}
W20260619 14:38:10.182003 9590 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:38:10.176625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58235 (local address 127.0.98.62:34179)
0619 14:38:10.176789 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:10.176794 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:10.176952 (+ 158us) server_negotiation.cc:407] Connection header received
0619 14:38:10.177082 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:10.177084 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:10.177132 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:10.177190 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:10.177563 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:10.178108 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:10.178850 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:10.179069 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:10.179155 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:10.181887 (+ 2732us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58235: BlockingRecv error: recv got EOF from 127.0.98.3:58235 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 19 14:38:10 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:38:10 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:38:10.828763 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:10.823351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34179 (local address 127.0.98.1:60633)
0619 14:38:10.823634 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:10.823648 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:10.823751 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:10.824047 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:10.824050 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:10.824261 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:38:10.824381 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:10.824389 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:10.825399 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:10.825402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:10.825984 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:10.825992 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:10.826385 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:10.826388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:10.826467 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:10.826944 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:38:10.826956 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:38:10.828627 (+ 1671us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":81,"threads_started":1}
W20260619 14:38:10.828902 9594 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:38:10.823497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60633 (local address 127.0.98.62:34179)
0619 14:38:10.823737 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:10.823740 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:10.823753 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:38:10.823842 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:10.823845 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:10.823894 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:10.823993 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:10.824515 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:10.825286 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:10.826078 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:10.826305 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:10.826382 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:10.828771 (+ 2389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60633: BlockingRecv error: recv got EOF from 127.0.98.1:60633 (error 108)
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":71,"threads_started":1}
Jun 19 14:38:10 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:38:10 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:38:11.004484 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:10.994487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34179 (local address 127.0.98.2:53669)
0619 14:38:10.994826 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:10.994840 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:10.994948 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:10.995224 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:10.995227 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:10.995414 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:38:10.995509 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:10.995515 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:10.996366 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:10.996369 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:10.997185 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:10.997196 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:11.000866 (+ 3670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:11.000869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:11.001007 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:11.001586 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:38:11.001625 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:38:11.004332 (+ 2707us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":88,"threads_started":1}
W20260619 14:38:11.004635 9594 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:38:10.994569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53669 (local address 127.0.98.62:34179)
0619 14:38:10.994723 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:10.994727 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:10.994896 (+ 169us) server_negotiation.cc:407] Connection header received
0619 14:38:10.995068 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:10.995071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:10.995115 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:10.995180 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:10.995620 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:10.996265 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:10.997321 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:10.997672 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:10.997766 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:11.004519 (+ 6753us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53669: BlockingRecv error: recv got EOF from 127.0.98.2:53669 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 19 14:38:11 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:38:11 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:38:11.187682 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:11.182634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34179 (local address 127.0.98.3:46667)
0619 14:38:11.182875 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:11.182887 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:11.182984 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:11.183183 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:11.183185 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:11.183343 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:38:11.183431 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:11.183436 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:11.184133 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:11.184136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:11.184698 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:11.184706 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:11.185226 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:11.185229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:11.185294 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:11.185833 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:38:11.185845 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:38:11.187565 (+ 1720us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":87,"threads_started":1}
W20260619 14:38:11.187793 9594 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:38:11.182681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46667 (local address 127.0.98.62:34179)
0619 14:38:11.182811 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:11.182816 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:11.182947 (+ 131us) server_negotiation.cc:407] Connection header received
0619 14:38:11.183048 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:11.183050 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:11.183097 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:11.183160 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:11.183529 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:11.184045 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:11.184843 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:11.185115 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:11.185206 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:11.187692 (+ 2486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46667: BlockingRecv error: recv got EOF from 127.0.98.3:46667 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 19 14:38:11 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:38:11 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:38:11.837498 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:11.829448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34179 (local address 127.0.98.1:55891)
0619 14:38:11.829687 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:11.829701 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:11.829796 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:11.830012 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:11.830014 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:11.830181 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:38:11.830280 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:11.830285 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:11.830967 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:11.830969 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:11.831541 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:11.831548 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:11.831900 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:11.831903 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:11.831960 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:11.835375 (+ 3415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:38:11.835387 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:38:11.837354 (+ 1967us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":75,"threads_started":1}
W20260619 14:38:11.837607 9598 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:38:11.829502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55891 (local address 127.0.98.62:34179)
0619 14:38:11.829721 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:11.829725 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:11.829739 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:38:11.829868 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:11.829870 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:11.829910 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:11.829989 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:11.830361 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:11.830887 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:11.831621 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:11.831827 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:11.831893 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:11.837495 (+ 5602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55891: BlockingRecv error: recv got EOF from 127.0.98.1:55891 (error 108)
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":64,"threads_started":1}
Jun 19 14:38:12 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:38:12 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:38:12.011700 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:12.006419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34179 (local address 127.0.98.2:55271)
0619 14:38:12.006761 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:12.006777 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:12.006889 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:12.007144 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:12.007146 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:12.007337 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:38:12.007425 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:12.007431 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:12.008163 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:12.008166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:12.008793 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:12.008802 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:12.009227 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:12.009230 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:12.009304 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:12.009784 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:38:12.009794 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:38:12.011586 (+ 1792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":98,"threads_started":1}
W20260619 14:38:12.011811 9598 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:38:12.006562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55271 (local address 127.0.98.62:34179)
0619 14:38:12.006696 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:12.006701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:12.006851 (+ 150us) server_negotiation.cc:407] Connection header received
0619 14:38:12.006997 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:12.007000 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:12.007043 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:12.007106 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:12.007543 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:12.008063 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:12.008905 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:12.009129 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:12.009215 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:12.011713 (+ 2498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55271: BlockingRecv error: recv got EOF from 127.0.98.2:55271 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Jun 19 14:38:12 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:38:12 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:38:12.193557 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:12.188334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34179 (local address 127.0.98.3:49547)
0619 14:38:12.188600 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:12.188613 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:12.188705 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:12.188947 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:12.188949 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:12.189115 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:38:12.189206 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:12.189211 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:12.189907 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:12.189911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:12.190677 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:12.190688 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:12.191168 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:12.191171 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:12.191240 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:12.191705 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:38:12.191720 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:38:12.193449 (+ 1729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":83,"threads_started":1}
W20260619 14:38:12.193689 9598 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:38:12.188402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49547 (local address 127.0.98.62:34179)
0619 14:38:12.188535 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:12.188538 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:12.188672 (+ 134us) server_negotiation.cc:407] Connection header received
0619 14:38:12.188804 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:12.188806 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:12.188850 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:12.188913 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:12.189286 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:12.189807 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:12.190793 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:12.191071 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:12.191150 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:12.193575 (+ 2425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49547: BlockingRecv error: recv got EOF from 127.0.98.3:49547 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 19 14:38:12 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:38:12 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:38:12.843603 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:12.838291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34179 (local address 127.0.98.1:41759)
0619 14:38:12.838553 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:12.838568 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:12.838650 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:12.838922 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:12.838925 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:12.839116 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:38:12.839214 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:12.839220 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:12.839986 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:12.839988 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:12.840635 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:12.840644 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:12.841190 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:12.841194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:12.841271 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:12.841741 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:38:12.841752 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:38:12.843461 (+ 1709us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":64,"threads_started":1}
W20260619 14:38:12.843709 9602 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:38:12.838370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41759 (local address 127.0.98.62:34179)
0619 14:38:12.838597 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:12.838600 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:12.838613 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:38:12.838720 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:12.838722 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:12.838770 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:12.838848 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:12.839349 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:12.839881 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:12.840789 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:12.841082 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:12.841184 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:12.843599 (+ 2415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41759: BlockingRecv error: recv got EOF from 127.0.98.1:41759 (error 108)
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":65,"threads_started":1}
Jun 19 14:38:13 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:38:13 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:38:13.017606 9603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:13.012510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34179 (local address 127.0.98.2:50447)
0619 14:38:13.012791 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:13.012806 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:13.012904 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:13.013142 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:13.013145 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:13.013343 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:38:13.013440 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:13.013446 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:13.014224 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:13.014227 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:13.014786 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:13.014794 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:13.015206 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:13.015210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:13.015295 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:13.015769 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:38:13.015780 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:38:13.017461 (+ 1681us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":84,"threads_started":1}
W20260619 14:38:13.017694 9602 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:38:13.012618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50447 (local address 127.0.98.62:34179)
0619 14:38:13.012801 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:13.012805 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:13.012873 (+ 68us) server_negotiation.cc:407] Connection header received
0619 14:38:13.012980 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:13.012982 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:13.013026 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:13.013102 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:13.013561 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:13.014112 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:13.014893 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:13.015110 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:13.015193 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:13.017597 (+ 2404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50447: BlockingRecv error: recv got EOF from 127.0.98.2:50447 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 19 14:38:13 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:38:13 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:38:13.199247 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:13.194254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34179 (local address 127.0.98.3:49059)
0619 14:38:13.194550 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:13.194564 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:13.194662 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:13.194928 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:13.194932 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:13.195102 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:38:13.195197 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:13.195203 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:13.195878 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:13.195881 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:13.196435 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:13.196442 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:13.196874 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:13.196876 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:13.196939 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:13.197449 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:38:13.197461 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:38:13.199103 (+ 1642us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":86,"threads_started":1}
W20260619 14:38:13.199332 9602 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:38:13.194414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49059 (local address 127.0.98.62:34179)
0619 14:38:13.194549 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:13.194552 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:13.194657 (+ 105us) server_negotiation.cc:407] Connection header received
0619 14:38:13.194743 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:13.194746 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:13.194792 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:13.194850 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:13.195289 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:13.195799 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:13.196545 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:13.196781 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:13.196859 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:13.199223 (+ 2364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49059: BlockingRecv error: recv got EOF from 127.0.98.3:49059 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 19 14:38:13 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:38:13 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:38:13.849229 9605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:13.844286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34179 (local address 127.0.98.1:33611)
0619 14:38:13.844622 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:13.844636 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:13.844767 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:13.845025 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:13.845028 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:13.845198 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:38:13.845296 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:13.845302 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:13.845990 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:13.845992 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:13.846540 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:13.846547 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:13.846901 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:13.846903 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:13.846965 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:13.847435 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:38:13.847447 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:38:13.849115 (+ 1668us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":93,"threads_started":1}
W20260619 14:38:13.849376 9606 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:38:13.844390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33611 (local address 127.0.98.62:34179)
0619 14:38:13.844623 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:13.844627 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:13.844702 (+ 75us) server_negotiation.cc:407] Connection header received
0619 14:38:13.844836 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:13.844838 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:13.844881 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:13.844954 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:13.845389 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:13.845916 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:13.846619 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:13.846827 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:13.846895 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:13.849245 (+ 2350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33611: BlockingRecv error: recv got EOF from 127.0.98.1:33611 (error 108)
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":56,"threads_started":1}
Jun 19 14:38:14 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:38:14 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:38:14.023252 9607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:14.018280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34179 (local address 127.0.98.2:58773)
0619 14:38:14.018517 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:14.018530 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:14.018623 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:14.018801 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:14.018803 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:14.018993 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:38:14.019086 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:14.019091 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:14.019821 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:14.019825 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:14.020413 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:14.020422 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:14.020849 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:14.020852 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:14.020914 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:14.021416 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:38:14.021428 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:38:14.023103 (+ 1675us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":68,"threads_started":1}
W20260619 14:38:14.023331 9606 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:38:14.018354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58773 (local address 127.0.98.62:34179)
0619 14:38:14.018514 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:14.018518 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:14.018578 (+ 60us) server_negotiation.cc:407] Connection header received
0619 14:38:14.018685 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:14.018687 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:14.018729 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:14.018790 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:14.019164 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:14.019687 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:14.020509 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:14.020747 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:14.020841 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:14.023236 (+ 2395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58773: BlockingRecv error: recv got EOF from 127.0.98.2:58773 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 19 14:38:14 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:38:14 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:38:14.205310 9608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:14.200025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34179 (local address 127.0.98.3:40913)
0619 14:38:14.200294 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:14.200308 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:14.200420 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:14.200687 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:14.200689 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:14.200878 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:38:14.200972 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:14.200977 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:14.201667 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:14.201670 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:14.202224 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:14.202231 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:14.202631 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:14.202634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:14.202718 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:14.203265 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:38:14.203277 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:38:14.205163 (+ 1886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":76,"threads_started":1}
W20260619 14:38:14.205391 9606 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:38:14.200100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40913 (local address 127.0.98.62:34179)
0619 14:38:14.200234 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:14.200239 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:14.200366 (+ 127us) server_negotiation.cc:407] Connection header received
0619 14:38:14.200500 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:14.200503 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:14.200555 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:14.200618 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:14.201062 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:14.201562 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:14.202310 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:14.202514 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:14.202582 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:14.205294 (+ 2712us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:40913: BlockingRecv error: recv got EOF from 127.0.98.3:40913 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 19 14:38:14 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:38:14 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:38:14.855491 9609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:14.850089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34179 (local address 127.0.98.1:51701)
0619 14:38:14.850349 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:14.850362 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:14.850462 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:14.850694 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:14.850696 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:14.850870 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:38:14.850985 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:14.850991 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:14.851779 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:14.851781 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:14.852360 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:14.852369 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:14.852829 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:14.852832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:14.852905 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:14.853459 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:38:14.853474 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:38:14.855378 (+ 1904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":76,"threads_started":1}
W20260619 14:38:14.855651 9610 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:38:14.850235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51701 (local address 127.0.98.62:34179)
0619 14:38:14.850471 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:14.850474 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:14.850515 (+ 41us) server_negotiation.cc:407] Connection header received
0619 14:38:14.850553 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:14.850556 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:14.850596 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:14.850667 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:14.851118 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:14.851663 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:14.852476 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:14.852703 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:14.852847 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:14.855517 (+ 2670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51701: BlockingRecv error: recv got EOF from 127.0.98.1:51701 (error 108)
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":71,"threads_started":1}
Jun 19 14:38:15 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:38:15 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:38:15.029511 9611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:15.024069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34179 (local address 127.0.98.2:37341)
0619 14:38:15.024378 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:15.024401 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:15.024519 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:15.024774 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:15.024777 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:15.025000 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:38:15.025112 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:15.025118 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:15.025988 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:15.025991 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:15.026599 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:15.026608 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:15.027009 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:15.027011 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:15.027067 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:15.027536 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:38:15.027548 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:38:15.029390 (+ 1842us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":106,"threads_started":1}
W20260619 14:38:15.029624 9610 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:38:15.024246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37341 (local address 127.0.98.62:34179)
0619 14:38:15.024379 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:15.024384 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:15.024467 (+ 83us) server_negotiation.cc:407] Connection header received
0619 14:38:15.024604 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:15.024608 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:15.024653 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:15.024754 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:15.025228 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:15.025879 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:15.026699 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:15.026923 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:15.027001 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:15.029529 (+ 2528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37341: BlockingRecv error: recv got EOF from 127.0.98.2:37341 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 19 14:38:15 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:38:15 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:38:15.211083 9612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:15.206091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34179 (local address 127.0.98.3:46599)
0619 14:38:15.206342 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:15.206355 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:15.206473 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:15.206682 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:15.206683 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:15.206852 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:38:15.206948 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:15.206952 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:15.207605 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:15.207607 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:15.208150 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:15.208157 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:15.208519 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:15.208521 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:15.208586 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:15.209056 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:38:15.209067 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:38:15.210971 (+ 1904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":81,"threads_started":1}
W20260619 14:38:15.211202 9610 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:38:15.206171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46599 (local address 127.0.98.62:34179)
0619 14:38:15.206311 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:15.206316 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:15.206436 (+ 120us) server_negotiation.cc:407] Connection header received
0619 14:38:15.206543 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:15.206545 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:15.206592 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:15.206658 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:15.207028 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:15.207529 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:15.208227 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:15.208426 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:15.208491 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:15.211097 (+ 2606us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46599: BlockingRecv error: recv got EOF from 127.0.98.3:46599 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 19 14:38:15 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:38:15 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:38:15.861393 9613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:15.856214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34179 (local address 127.0.98.1:43023)
0619 14:38:15.856475 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:15.856490 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:15.856610 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:15.856907 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:15.856910 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:15.857102 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:38:15.857195 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:15.857201 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:15.857987 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:15.857991 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:15.858549 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:15.858557 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:15.858982 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:15.858986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:15.859065 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:15.859541 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:38:15.859553 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:38:15.861248 (+ 1695us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":77,"threads_started":1}
W20260619 14:38:15.861527 9614 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:38:15.856250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43023 (local address 127.0.98.62:34179)
0619 14:38:15.856491 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:15.856496 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:15.856562 (+ 66us) server_negotiation.cc:407] Connection header received
0619 14:38:15.856696 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:15.856699 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:15.856776 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:15.856861 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:15.857306 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:15.857881 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:15.858650 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:15.858887 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:15.858971 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:15.861413 (+ 2442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43023: BlockingRecv error: recv got EOF from 127.0.98.1:43023 (error 108)
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":70,"threads_started":1}
Jun 19 14:38:16 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:38:16 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:38:16.035633 9615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:16.030292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34179 (local address 127.0.98.2:39967)
0619 14:38:16.030602 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:16.030617 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:16.030723 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:16.030952 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:16.030954 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:16.031123 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:38:16.031212 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:16.031217 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:16.031949 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:16.031951 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:16.032647 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:16.032657 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:16.033183 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:16.033187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:16.033269 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:16.033716 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:38:16.033728 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:38:16.035488 (+ 1760us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":113,"threads_started":1}
W20260619 14:38:16.035728 9614 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:38:16.030452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39967 (local address 127.0.98.62:34179)
0619 14:38:16.030598 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:16.030602 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:16.030681 (+ 79us) server_negotiation.cc:407] Connection header received
0619 14:38:16.030797 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:16.030800 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:16.030842 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:16.030928 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:16.031298 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:16.031827 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:16.032773 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:16.033068 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:16.033168 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:16.035618 (+ 2450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39967: BlockingRecv error: recv got EOF from 127.0.98.2:39967 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 14:38:16 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:38:16 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:38:16.216801 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:16.211806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34179 (local address 127.0.98.3:49393)
0619 14:38:16.212075 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:16.212089 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:16.212184 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:16.212424 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:16.212426 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:16.212595 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:38:16.212685 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:16.212690 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:16.213439 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:16.213441 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:16.213995 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:16.214002 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:16.214422 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:16.214424 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:16.214481 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:16.214928 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:38:16.214939 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:38:16.216631 (+ 1692us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":105,"threads_started":1}
W20260619 14:38:16.216889 9614 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:38:16.211850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49393 (local address 127.0.98.62:34179)
0619 14:38:16.211985 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:16.211990 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:16.212180 (+ 190us) server_negotiation.cc:407] Connection header received
0619 14:38:16.212250 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:16.212252 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:16.212302 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:16.212397 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:16.212791 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:16.213354 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:16.214095 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:16.214331 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:16.214414 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:16.216789 (+ 2375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49393: BlockingRecv error: recv got EOF from 127.0.98.3:49393 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 19 14:38:16 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:38:16 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:38:16.869976 9617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:16.862137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34179 (local address 127.0.98.1:54331)
0619 14:38:16.862388 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:16.862401 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:16.862487 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:16.862736 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:16.862738 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:16.862929 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:38:16.863030 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:16.863036 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:16.863751 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:16.863754 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:16.864319 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:16.864327 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:16.864786 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:16.864789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:16.864860 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:16.868101 (+ 3241us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:38:16.868114 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:38:16.869830 (+ 1716us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":83,"threads_started":1}
W20260619 14:38:16.870136 9618 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:38:16.862254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54331 (local address 127.0.98.62:34179)
0619 14:38:16.862510 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:16.862514 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:16.862527 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:38:16.862561 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:16.862563 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:16.862604 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:16.862693 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:16.863128 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:16.863647 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:16.864429 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:16.864671 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:16.864786 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:16.869996 (+ 5210us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54331: BlockingRecv error: recv got EOF from 127.0.98.1:54331 (error 108)
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":63,"threads_started":1}
Jun 19 14:38:17 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:38:17 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:38:17.041617 9619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:17.036376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34179 (local address 127.0.98.2:57763)
0619 14:38:17.036664 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:17.036679 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:17.036820 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:17.037065 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:17.037068 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:17.037261 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:38:17.037379 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:17.037386 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:17.038152 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:17.038155 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:17.038798 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:17.038808 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:17.039242 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:17.039246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:17.039322 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:17.039774 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:38:17.039786 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:38:17.041487 (+ 1701us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":84,"threads_started":1}
W20260619 14:38:17.041719 9618 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:38:17.036492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57763 (local address 127.0.98.62:34179)
0619 14:38:17.036649 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:17.036653 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:17.036749 (+ 96us) server_negotiation.cc:407] Connection header received
0619 14:38:17.036921 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:17.036923 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:17.036968 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:17.037033 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:17.037492 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:17.038050 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:17.038903 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:17.039143 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:17.039238 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:17.041616 (+ 2378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57763: BlockingRecv error: recv got EOF from 127.0.98.2:57763 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 19 14:38:17 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:38:17 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:38:17.222362 9620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:17.217492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34179 (local address 127.0.98.3:40809)
0619 14:38:17.217765 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:17.217778 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:17.217872 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:17.218073 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:17.218075 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:17.218241 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:38:17.218331 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:17.218336 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:17.219080 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:17.219083 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:17.219659 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:17.219667 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:17.220058 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:17.220060 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:17.220117 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:17.220613 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:38:17.220624 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:38:17.222212 (+ 1588us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":80,"threads_started":1}
W20260619 14:38:17.222489 9618 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:38:17.217578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40809 (local address 127.0.98.62:34179)
0619 14:38:17.217716 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:17.217720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:17.217835 (+ 115us) server_negotiation.cc:407] Connection header received
0619 14:38:17.217935 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:17.217937 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:17.217980 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:17.218047 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:17.218457 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:17.218984 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:17.219751 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:17.219976 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:17.220050 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:17.222377 (+ 2327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:40809: BlockingRecv error: recv got EOF from 127.0.98.3:40809 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Jun 19 14:38:17 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:38:17 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:38:17.875906 9621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:17.870733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34179 (local address 127.0.98.1:47311)
0619 14:38:17.870962 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:17.870976 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:17.871054 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:17.871328 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:17.871330 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:17.871502 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:38:17.871596 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:17.871602 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:17.872426 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:17.872429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:17.873059 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:17.873070 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:17.873524 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:17.873526 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:17.873590 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:17.874102 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:38:17.874115 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:38:17.875773 (+ 1658us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":65,"threads_started":1}
W20260619 14:38:17.876047 9622 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:38:17.870805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47311 (local address 127.0.98.62:34179)
0619 14:38:17.871001 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:17.871004 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:17.871016 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:38:17.871124 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:17.871126 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:17.871184 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:17.871258 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:17.871731 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:17.872304 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:17.873179 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:17.873428 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:17.873510 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:17.875926 (+ 2416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47311: BlockingRecv error: recv got EOF from 127.0.98.1:47311 (error 108)
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":61,"threads_started":1}
Jun 19 14:38:18 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:38:18 dist-test-slave-32hj krb5kdc[8564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:38:18.047518 9623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:18.042418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34179 (local address 127.0.98.2:54629)
0619 14:38:18.042673 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:18.042685 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:18.042787 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:18.043024 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:18.043027 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:18.043212 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:38:18.043301 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:18.043307 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:18.044033 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:18.044037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:18.044706 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:18.044750 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:18.045196 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:18.045199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:18.045287 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:18.045730 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:38:18.045741 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:38:18.047405 (+ 1664us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34179: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":80,"threads_started":1}
W20260619 14:38:18.047639 9622 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:38:18.042465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54629 (local address 127.0.98.62:34179)
0619 14:38:18.042617 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:18.042622 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:18.042755 (+ 133us) server_negotiation.cc:407] Connection header received
0619 14:38:18.042873 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:18.042877 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:18.042925 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:18.042989 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:18.043413 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:18.043934 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:18.044871 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:18.045098 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:18.045187 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:18.047538 (+ 2351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54629: BlockingRecv error: recv got EOF from 127.0.98.2:54629 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 14:38:18.121538 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 9147
I20260619 14:38:18.126106 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 9299
I20260619 14:38:18.130632 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 9431
I20260619 14:38:18.134855 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 9077
2026-06-19T14:38:18Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17834 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-19T14:38:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:38:18Z Disabled control of system clock
I20260619 14:38:18.162555 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41753
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41753
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 14:38:18.269743 9627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:38:18.270004 9627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:38:18.270035 9627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:38:18.273962 9627 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:38:18.274021 9627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:38:18.274035 9627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:38:18.274050 9627 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:38:18.274063 9627 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:38:18.278759 9627 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41753
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:38:18.279776 9627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:38:18.281102 9627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:38:18.289503 9635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:38:18.289491 9632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:38:18.289556 9627 server_base.cc:1034] running on GCE node
W20260619 14:38:18.289513 9633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:38:18.290336 9627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:38:18.291419 9627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:38:18.292586 9627 hybrid_clock.cc:648] HybridClock initialized: now 1781879898292569 us; error 30 us; skew 500 ppm
I20260619 14:38:18.294628 9627 webserver.cc:460] Webserver started at http://127.0.98.62:37795/ using document root <none> and password file <none>
I20260619 14:38:18.295219 9627 fs_manager.cc:359] Metadata directory not provided
I20260619 14:38:18.295295 9627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:38:18.295499 9627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:38:18.297645 9627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/master-0/data/instance:
uuid: "68bffa75d71340848265c0e19bab8087"
format_stamp: "Formatted at 2026-06-19 14:38:18 on dist-test-slave-32hj"
server_key: "db1ccfdfe26fce0ffb62bd48afe5bb7e"
server_key_iv: "a5fd140edbf570d4147b539c0dd8716b"
server_key_version: "encryptionkey@0"
I20260619 14:38:18.298218 9627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/master-0/wal/instance:
uuid: "68bffa75d71340848265c0e19bab8087"
format_stamp: "Formatted at 2026-06-19 14:38:18 on dist-test-slave-32hj"
server_key: "db1ccfdfe26fce0ffb62bd48afe5bb7e"
server_key_iv: "a5fd140edbf570d4147b539c0dd8716b"
server_key_version: "encryptionkey@0"
I20260619 14:38:18.301936 9627 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:38:18.304226 9641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:38:18.305449 9627 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:38:18.305560 9627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/master-0/wal
uuid: "68bffa75d71340848265c0e19bab8087"
format_stamp: "Formatted at 2026-06-19 14:38:18 on dist-test-slave-32hj"
server_key: "db1ccfdfe26fce0ffb62bd48afe5bb7e"
server_key_iv: "a5fd140edbf570d4147b539c0dd8716b"
server_key_version: "encryptionkey@0"
I20260619 14:38:18.305653 9627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:38:18.321089 9627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:38:18.321750 9627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:38:18.321928 9627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:38:18.328240 9627 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41753
I20260619 14:38:18.328253 9693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41753 every 8 connection(s)
I20260619 14:38:18.329171 9627 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:38:18.332082 9694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:38:18.337343 9694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68bffa75d71340848265c0e19bab8087: Bootstrap starting.
I20260619 14:38:18.337785 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 9627
I20260619 14:38:18.337900 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/master-0/wal/instance
I20260619 14:38:18.338178 392 external_mini_cluster.cc:1428] Setting key f136e5f5c845e425d148976285cf9154
I20260619 14:38:18.340003 9694 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68bffa75d71340848265c0e19bab8087: Neither blocks nor log segments found. Creating new log.
I20260619 14:38:18.340693 9694 log.cc:824] T 00000000000000000000000000000000 P 68bffa75d71340848265c0e19bab8087: Log is configured to *not* fsync() on all Append() calls
I20260619 14:38:18.342713 9694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68bffa75d71340848265c0e19bab8087: No bootstrap required, opened a new log
I20260619 14:38:18.346200 9694 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68bffa75d71340848265c0e19bab8087 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68bffa75d71340848265c0e19bab8087" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41753 } }
I20260619 14:38:18.346490 9694 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68bffa75d71340848265c0e19bab8087 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:38:18.346554 9694 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68bffa75d71340848265c0e19bab8087 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68bffa75d71340848265c0e19bab8087, State: Initialized, Role: FOLLOWER
I20260619 14:38:18.347008 9694 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68bffa75d71340848265c0e19bab8087 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68bffa75d71340848265c0e19bab8087" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41753 } }
I20260619 14:38:18.347146 9694 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68bffa75d71340848265c0e19bab8087 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:38:18.347214 9694 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68bffa75d71340848265c0e19bab8087 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:38:18.347246 9697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:18.339508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41206 (local address 127.0.98.62:41753)
0619 14:38:18.340164 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:18.340179 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:18.340214 (+ 35us) server_negotiation.cc:407] Connection header received
0619 14:38:18.340876 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:18.340896 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:18.341366 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:18.341709 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:18.342130 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:18.343127 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:18.343999 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:18.344481 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:18.344627 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:18.345464 (+ 837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:38:18.345469 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:38:18.345481 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:38:18.345871 (+ 390us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:38:18.345927 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:38:18.345947 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:38:18.346032 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 14:38:18.346181 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 14:38:18.346432 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":123,"threads_started":1}
I20260619 14:38:18.347316 9694 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68bffa75d71340848265c0e19bab8087 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:38:18.348223 9694 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68bffa75d71340848265c0e19bab8087 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68bffa75d71340848265c0e19bab8087" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41753 } }
I20260619 14:38:18.348575 9694 leader_election.cc:302] T 00000000000000000000000000000000 P 68bffa75d71340848265c0e19bab8087 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68bffa75d71340848265c0e19bab8087; no voters:
I20260619 14:38:18.348966 9694 leader_election.cc:288] T 00000000000000000000000000000000 P 68bffa75d71340848265c0e19bab8087 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:38:18.349056 9699 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68bffa75d71340848265c0e19bab8087 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:38:18.349323 9699 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68bffa75d71340848265c0e19bab8087 [term 1 LEADER]: Becoming Leader. State: Replica: 68bffa75d71340848265c0e19bab8087, State: Running, Role: LEADER
I20260619 14:38:18.349761 9699 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68bffa75d71340848265c0e19bab8087 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68bffa75d71340848265c0e19bab8087" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41753 } }
I20260619 14:38:18.350287 9694 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68bffa75d71340848265c0e19bab8087 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:38:18.351410 9700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68bffa75d71340848265c0e19bab8087 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68bffa75d71340848265c0e19bab8087. Latest consensus state: current_term: 1 leader_uuid: "68bffa75d71340848265c0e19bab8087" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68bffa75d71340848265c0e19bab8087" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41753 } } }
I20260619 14:38:18.351425 9701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68bffa75d71340848265c0e19bab8087 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68bffa75d71340848265c0e19bab8087" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68bffa75d71340848265c0e19bab8087" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41753 } } }
I20260619 14:38:18.351614 9701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68bffa75d71340848265c0e19bab8087 [sys.catalog]: This master's current role is: LEADER
I20260619 14:38:18.351926 9700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68bffa75d71340848265c0e19bab8087 [sys.catalog]: This master's current role is: LEADER
I20260619 14:38:18.352020 9708 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:38:18.354851 9708 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:38:18.360342 9708 catalog_manager.cc:1347] Generated new cluster ID: c70fbae0e05648fb937957beb367c6dc
I20260619 14:38:18.360419 9708 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:38:18.374881 9708 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:38:18.376057 9708 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:38:18.385990 9708 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68bffa75d71340848265c0e19bab8087: Generated new TSK 0
I20260619 14:38:18.386663 9708 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:38:18.394554 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41753
--builtin_ntp_servers=127.0.98.20:37893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260619 14:38:18.502200 9718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:38:18.502451 9718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:38:18.502487 9718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:38:18.506397 9718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:38:18.506444 9718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:38:18.506496 9718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:38:18.511125 9718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:38:18.512180 9718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:38:18.513527 9718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:38:18.522496 9724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:38:18.522603 9718 server_base.cc:1034] running on GCE node
W20260619 14:38:18.522527 9723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:38:18.522532 9726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:38:18.523087 9718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:38:18.523725 9718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:38:18.524891 9718 hybrid_clock.cc:648] HybridClock initialized: now 1781879898524871 us; error 31 us; skew 500 ppm
I20260619 14:38:18.526623 9718 webserver.cc:460] Webserver started at http://127.0.98.1:43863/ using document root <none> and password file <none>
I20260619 14:38:18.527177 9718 fs_manager.cc:359] Metadata directory not provided
I20260619 14:38:18.527244 9718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:38:18.527434 9718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:38:18.529475 9718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/ts-0/data/instance:
uuid: "ba7e5048edaf430895712a81e8bd779b"
format_stamp: "Formatted at 2026-06-19 14:38:18 on dist-test-slave-32hj"
server_key: "42850af20a105394c4e19615e0918690"
server_key_iv: "fc13a6e7eb6f976f40e8aba10276138c"
server_key_version: "encryptionkey@0"
I20260619 14:38:18.529999 9718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ba7e5048edaf430895712a81e8bd779b"
format_stamp: "Formatted at 2026-06-19 14:38:18 on dist-test-slave-32hj"
server_key: "42850af20a105394c4e19615e0918690"
server_key_iv: "fc13a6e7eb6f976f40e8aba10276138c"
server_key_version: "encryptionkey@0"
I20260619 14:38:18.533536 9718 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:38:18.535944 9732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:38:18.537160 9718 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:38:18.537261 9718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/ts-0/wal
uuid: "ba7e5048edaf430895712a81e8bd779b"
format_stamp: "Formatted at 2026-06-19 14:38:18 on dist-test-slave-32hj"
server_key: "42850af20a105394c4e19615e0918690"
server_key_iv: "fc13a6e7eb6f976f40e8aba10276138c"
server_key_version: "encryptionkey@0"
I20260619 14:38:18.537372 9718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:38:18.557653 9718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:38:18.558360 9718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:38:18.558531 9718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:38:18.559103 9718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:38:18.560015 9718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:38:18.560062 9718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:38:18.560114 9718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:38:18.560143 9718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:38:18.568221 9718 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36345
I20260619 14:38:18.568267 9845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36345 every 8 connection(s)
I20260619 14:38:18.569131 9718 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:38:18.570834 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 9718
I20260619 14:38:18.570945 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:38:18.571234 392 external_mini_cluster.cc:1428] Setting key 68af20d8203a79beeecbbc3fcabbacba
I20260619 14:38:18.573401 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41753
--builtin_ntp_servers=127.0.98.20:37893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 14:38:18.579140 9697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:18.570942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55783 (local address 127.0.98.62:41753)
0619 14:38:18.571071 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:18.571076 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:18.572280 (+ 1204us) server_negotiation.cc:407] Connection header received
0619 14:38:18.573989 (+ 1709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:18.573991 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:18.574033 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:18.574101 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:18.575184 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:18.575955 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:18.577139 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:18.577491 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:18.577599 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:18.578662 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:38:18.578667 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:38:18.578670 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:38:18.578684 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:38:18.578708 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:38:18.578719 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:38:18.578805 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 14:38:18.578922 (+ 117us) server_negotiation.cc:299] Negotiation successful
0619 14:38:18.578988 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 14:38:18.579921 9848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:18.571386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41753 (local address 127.0.98.1:55783)
0619 14:38:18.572140 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:18.572182 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:18.573759 (+ 1577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:18.574286 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:18.574295 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:18.574417 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:38:18.575038 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:18.575052 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:18.576068 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:18.576070 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:18.577024 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:18.577033 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:18.577637 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:18.577640 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:18.577786 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:18.578407 (+ 621us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:38:18.578409 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:38:18.578448 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:38:18.578510 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:38:18.578799 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:38:18.578804 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:38:18.578807 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 14:38:18.578896 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 14:38:18.579110 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":145,"threads_started":1}
I20260619 14:38:18.581095 9846 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41753
I20260619 14:38:18.581354 9846 heartbeater.cc:464] Registering TS with master...
I20260619 14:38:18.582013 9846 heartbeater.cc:511] Master 127.0.98.62:41753 requested a full tablet report, sending...
I20260619 14:38:18.583606 9658 ts_manager.cc:194] Registered new tserver with Master: ba7e5048edaf430895712a81e8bd779b (127.0.98.1:36345)
I20260619 14:38:18.584755 9658 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:55783
W20260619 14:38:18.681416 9849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:38:18.681684 9849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:38:18.681720 9849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:38:18.685721 9849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:38:18.685781 9849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:38:18.685832 9849 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:38:18.690505 9849 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9849
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:38:18.691560 9849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:38:18.692989 9849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:38:18.701939 9857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:38:18.701967 9855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:38:18.701984 9854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:38:18.702351 9849 server_base.cc:1034] running on GCE node
I20260619 14:38:18.702792 9849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:38:18.703395 9849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:38:18.704567 9849 hybrid_clock.cc:648] HybridClock initialized: now 1781879898704544 us; error 36 us; skew 500 ppm
I20260619 14:38:18.706378 9849 webserver.cc:460] Webserver started at http://127.0.98.2:43543/ using document root <none> and password file <none>
I20260619 14:38:18.706912 9849 fs_manager.cc:359] Metadata directory not provided
I20260619 14:38:18.706979 9849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:38:18.707158 9849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:38:18.709179 9849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/ts-1/data/instance:
uuid: "96dbdc6d7518472ab2e9aaf530632398"
format_stamp: "Formatted at 2026-06-19 14:38:18 on dist-test-slave-32hj"
server_key: "707763f13875f062bf5e332b66087121"
server_key_iv: "12a28d2e28fc2dfec8781976fb02e1d4"
server_key_version: "encryptionkey@0"
I20260619 14:38:18.709707 9849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/ts-1/wal/instance:
uuid: "96dbdc6d7518472ab2e9aaf530632398"
format_stamp: "Formatted at 2026-06-19 14:38:18 on dist-test-slave-32hj"
server_key: "707763f13875f062bf5e332b66087121"
server_key_iv: "12a28d2e28fc2dfec8781976fb02e1d4"
server_key_version: "encryptionkey@0"
I20260619 14:38:18.713153 9849 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:38:18.715585 9863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:38:18.716845 9849 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:38:18.716954 9849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/ts-1/wal
uuid: "96dbdc6d7518472ab2e9aaf530632398"
format_stamp: "Formatted at 2026-06-19 14:38:18 on dist-test-slave-32hj"
server_key: "707763f13875f062bf5e332b66087121"
server_key_iv: "12a28d2e28fc2dfec8781976fb02e1d4"
server_key_version: "encryptionkey@0"
I20260619 14:38:18.717051 9849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:38:18.732093 9849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:38:18.732759 9849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:38:18.732924 9849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:38:18.733459 9849 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:38:18.734316 9849 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:38:18.734370 9849 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:38:18.734421 9849 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:38:18.734449 9849 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:38:18.742549 9849 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38507
I20260619 14:38:18.742617 9976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38507 every 8 connection(s)
I20260619 14:38:18.743430 9849 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:38:18.749565 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 9849
I20260619 14:38:18.749675 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:38:18.749982 392 external_mini_cluster.cc:1428] Setting key 5a5d49db125fda48957419014c225b0b
I20260619 14:38:18.751945 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41753
--builtin_ntp_servers=127.0.98.20:37893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 14:38:18.752545 9697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:18.745148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40785 (local address 127.0.98.62:41753)
0619 14:38:18.745293 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:18.745298 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:18.746380 (+ 1082us) server_negotiation.cc:407] Connection header received
0619 14:38:18.747645 (+ 1265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:18.747647 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:18.747690 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:18.747761 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:18.748585 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:18.749142 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:18.750603 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:18.750811 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:18.750904 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:18.752052 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:38:18.752055 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:38:18.752058 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:38:18.752071 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:38:18.752092 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:38:18.752103 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:38:18.752169 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 14:38:18.752347 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 14:38:18.752405 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":53}
I20260619 14:38:18.753347 9979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:18.745567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41753 (local address 127.0.98.2:40785)
0619 14:38:18.746214 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:18.746257 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:18.747472 (+ 1215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:18.747913 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:18.747919 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:18.747998 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:38:18.748479 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:18.748491 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:18.749236 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:18.749238 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:18.750491 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:18.750501 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:18.750953 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:18.750957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:18.751168 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:18.751775 (+ 607us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:38:18.751776 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:38:18.751822 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:38:18.751901 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:38:18.752177 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:38:18.752185 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:38:18.752191 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 14:38:18.752324 (+ 133us) client_negotiation.cc:240] Negotiation successful
0619 14:38:18.752543 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":111,"threads_started":1}
I20260619 14:38:18.754526 9977 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41753
I20260619 14:38:18.754745 9977 heartbeater.cc:464] Registering TS with master...
I20260619 14:38:18.755230 9977 heartbeater.cc:511] Master 127.0.98.62:41753 requested a full tablet report, sending...
I20260619 14:38:18.756268 9658 ts_manager.cc:194] Registered new tserver with Master: 96dbdc6d7518472ab2e9aaf530632398 (127.0.98.2:38507)
I20260619 14:38:18.756858 9658 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:40785
W20260619 14:38:18.859614 9980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:38:18.859890 9980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:38:18.859943 9980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:38:18.864046 9980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:38:18.864099 9980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:38:18.864172 9980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:38:18.868878 9980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:38:18.869899 9980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:38:18.871224 9980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:38:18.880641 9985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:38:18.880650 9988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:38:18.880914 9980 server_base.cc:1034] running on GCE node
W20260619 14:38:18.880918 9986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:38:18.881366 9980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:38:18.881969 9980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:38:18.883157 9980 hybrid_clock.cc:648] HybridClock initialized: now 1781879898883137 us; error 31 us; skew 500 ppm
I20260619 14:38:18.884996 9980 webserver.cc:460] Webserver started at http://127.0.98.3:34769/ using document root <none> and password file <none>
I20260619 14:38:18.885535 9980 fs_manager.cc:359] Metadata directory not provided
I20260619 14:38:18.885596 9980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:38:18.885785 9980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:38:18.887900 9980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/ts-2/data/instance:
uuid: "5c75bbaaf1444743b2eb1e49c27456f5"
format_stamp: "Formatted at 2026-06-19 14:38:18 on dist-test-slave-32hj"
server_key: "e28f4487dc5f56de27084d5627a250d0"
server_key_iv: "49941aca603fdfa9e76ef27d868794f2"
server_key_version: "encryptionkey@0"
I20260619 14:38:18.888432 9980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5c75bbaaf1444743b2eb1e49c27456f5"
format_stamp: "Formatted at 2026-06-19 14:38:18 on dist-test-slave-32hj"
server_key: "e28f4487dc5f56de27084d5627a250d0"
server_key_iv: "49941aca603fdfa9e76ef27d868794f2"
server_key_version: "encryptionkey@0"
I20260619 14:38:18.891935 9980 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 14:38:18.894414 9994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:38:18.895682 9980 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 14:38:18.895778 9980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/ts-2/wal
uuid: "5c75bbaaf1444743b2eb1e49c27456f5"
format_stamp: "Formatted at 2026-06-19 14:38:18 on dist-test-slave-32hj"
server_key: "e28f4487dc5f56de27084d5627a250d0"
server_key_iv: "49941aca603fdfa9e76ef27d868794f2"
server_key_version: "encryptionkey@0"
I20260619 14:38:18.895889 9980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:38:18.920367 9980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:38:18.921175 9980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:38:18.921365 9980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:38:18.921983 9980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:38:18.922930 9980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:38:18.922981 9980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:38:18.923027 9980 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:38:18.923062 9980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:38:18.931743 9980 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38753
I20260619 14:38:18.931767 10107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38753 every 8 connection(s)
I20260619 14:38:18.932691 9980 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:38:18.938228 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 9980
I20260619 14:38:18.938356 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781879837648036-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:38:18.938632 392 external_mini_cluster.cc:1428] Setting key c8a56eadf6757cf40d22677c0d887afa
I20260619 14:38:18.942401 9697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:18.934455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58953 (local address 127.0.98.62:41753)
0619 14:38:18.934596 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:18.934601 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:18.935712 (+ 1111us) server_negotiation.cc:407] Connection header received
0619 14:38:18.937158 (+ 1446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:18.937162 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:18.937208 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:18.937306 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:18.938247 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:18.938969 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:18.940555 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:18.940884 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:18.940964 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:18.942053 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:38:18.942056 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:38:18.942058 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:38:18.942068 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:38:18.942086 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:38:18.942093 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:38:18.942145 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 14:38:18.942249 (+ 104us) server_negotiation.cc:299] Negotiation successful
0619 14:38:18.942295 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 14:38:18.943214 10110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:18.934863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41753 (local address 127.0.98.3:58953)
0619 14:38:18.935560 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:18.935605 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:18.936939 (+ 1334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:18.937485 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:18.937492 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:18.937576 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:38:18.938068 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:18.938081 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:18.939114 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:18.939118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:18.940384 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:18.940396 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:18.941018 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:18.941021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:18.941171 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:18.941822 (+ 651us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:38:18.941824 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:38:18.941866 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:38:18.941928 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:38:18.942147 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:38:18.942151 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:38:18.942155 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 14:38:18.942247 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 14:38:18.942458 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":107,"threads_started":1}
I20260619 14:38:18.944330 10108 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41753
I20260619 14:38:18.944555 10108 heartbeater.cc:464] Registering TS with master...
I20260619 14:38:18.945078 10108 heartbeater.cc:511] Master 127.0.98.62:41753 requested a full tablet report, sending...
I20260619 14:38:18.946018 9658 ts_manager.cc:194] Registered new tserver with Master: 5c75bbaaf1444743b2eb1e49c27456f5 (127.0.98.3:38753)
I20260619 14:38:18.946537 9658 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:58953
I20260619 14:38:18.952286 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 14:38:18.954931 9697 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:38:18.954072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41210 (local address 127.0.98.62:41753)
0619 14:38:18.954185 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:18.954189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:18.954377 (+ 188us) server_negotiation.cc:407] Connection header received
0619 14:38:18.954490 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:18.954493 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:18.954531 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:18.954593 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:18.954840 (+ 247us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41210: BlockingRecv error: recv got EOF from 127.0.0.1:41210 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 14:38:18.955408 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 9718
I20260619 14:38:18.960202 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 9849
I20260619 14:38:18.964502 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 9980
I20260619 14:38:18.968743 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 9627
2026-06-19T14:38:18Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (832 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-19T14:38:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:38:18Z Disabled control of system clock
I20260619 14:38:18.994791 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46195
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46195
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 14:38:19.100699 10119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:38:19.100970 10119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:38:19.101004 10119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:38:19.104851 10119 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:38:19.104902 10119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:38:19.104916 10119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:38:19.104930 10119 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:38:19.104943 10119 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:38:19.109593 10119 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46195
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:38:19.110556 10119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:38:19.111860 10119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:38:19.119920 10125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:38:19.119948 10124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:38:19.120074 10119 server_base.cc:1034] running on GCE node
W20260619 14:38:19.119935 10127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:38:19.120810 10119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:38:19.121901 10119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:38:19.123063 10119 hybrid_clock.cc:648] HybridClock initialized: now 1781879899123035 us; error 40 us; skew 500 ppm
I20260619 14:38:19.124991 10119 webserver.cc:460] Webserver started at http://127.0.98.62:36759/ using document root <none> and password file <none>
I20260619 14:38:19.125586 10119 fs_manager.cc:359] Metadata directory not provided
I20260619 14:38:19.125651 10119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:38:19.125844 10119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:38:19.127851 10119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/master-0/data/instance:
uuid: "ffc098b71ea7489f96e5e2ca00352ce1"
format_stamp: "Formatted at 2026-06-19 14:38:19 on dist-test-slave-32hj"
server_key: "313640e93d9d56de707fdb4fcba9f9f9"
server_key_iv: "9714f3017f6aed8a741aac220d5d5258"
server_key_version: "encryptionkey@0"
I20260619 14:38:19.128407 10119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/master-0/wal/instance:
uuid: "ffc098b71ea7489f96e5e2ca00352ce1"
format_stamp: "Formatted at 2026-06-19 14:38:19 on dist-test-slave-32hj"
server_key: "313640e93d9d56de707fdb4fcba9f9f9"
server_key_iv: "9714f3017f6aed8a741aac220d5d5258"
server_key_version: "encryptionkey@0"
I20260619 14:38:19.132031 10119 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 14:38:19.134685 10133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:38:19.136062 10119 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:38:19.136175 10119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/master-0/wal
uuid: "ffc098b71ea7489f96e5e2ca00352ce1"
format_stamp: "Formatted at 2026-06-19 14:38:19 on dist-test-slave-32hj"
server_key: "313640e93d9d56de707fdb4fcba9f9f9"
server_key_iv: "9714f3017f6aed8a741aac220d5d5258"
server_key_version: "encryptionkey@0"
I20260619 14:38:19.136308 10119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:38:19.141228 10119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:38:19.141856 10119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:38:19.142015 10119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:38:19.148712 10119 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46195
I20260619 14:38:19.148774 10185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46195 every 8 connection(s)
I20260619 14:38:19.149659 10119 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:38:19.152781 10186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:38:19.157502 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 10119
I20260619 14:38:19.157620 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/master-0/wal/instance
I20260619 14:38:19.157912 392 external_mini_cluster.cc:1428] Setting key 1b1c6ac317b77cf45a55f165e183d3d3
I20260619 14:38:19.158432 10186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffc098b71ea7489f96e5e2ca00352ce1: Bootstrap starting.
I20260619 14:38:19.160768 10186 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ffc098b71ea7489f96e5e2ca00352ce1: Neither blocks nor log segments found. Creating new log.
I20260619 14:38:19.161562 10186 log.cc:824] T 00000000000000000000000000000000 P ffc098b71ea7489f96e5e2ca00352ce1: Log is configured to *not* fsync() on all Append() calls
I20260619 14:38:19.163599 10186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffc098b71ea7489f96e5e2ca00352ce1: No bootstrap required, opened a new log
I20260619 14:38:19.163653 10189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:19.159329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58960 (local address 127.0.98.62:46195)
0619 14:38:19.159938 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:19.159954 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:19.159989 (+ 35us) server_negotiation.cc:407] Connection header received
0619 14:38:19.160615 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:19.160633 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:19.161055 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:19.161343 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:19.162163 (+ 820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:38:19.162169 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:38:19.162211 (+ 42us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:38:19.162225 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:38:19.162296 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:38:19.162324 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:38:19.162416 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 14:38:19.162544 (+ 128us) server_negotiation.cc:299] Negotiation successful
0619 14:38:19.162786 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":106,"threads_started":1}
I20260619 14:38:19.167279 10186 raft_consensus.cc:348] T 00000000000000000000000000000000 P ffc098b71ea7489f96e5e2ca00352ce1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc098b71ea7489f96e5e2ca00352ce1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46195 } }
I20260619 14:38:19.167492 10186 raft_consensus.cc:374] T 00000000000000000000000000000000 P ffc098b71ea7489f96e5e2ca00352ce1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:38:19.167541 10186 raft_consensus.cc:729] T 00000000000000000000000000000000 P ffc098b71ea7489f96e5e2ca00352ce1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffc098b71ea7489f96e5e2ca00352ce1, State: Initialized, Role: FOLLOWER
I20260619 14:38:19.167924 10186 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffc098b71ea7489f96e5e2ca00352ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc098b71ea7489f96e5e2ca00352ce1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46195 } }
I20260619 14:38:19.168047 10186 raft_consensus.cc:388] T 00000000000000000000000000000000 P ffc098b71ea7489f96e5e2ca00352ce1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:38:19.168108 10186 raft_consensus.cc:482] T 00000000000000000000000000000000 P ffc098b71ea7489f96e5e2ca00352ce1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:38:19.168202 10186 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ffc098b71ea7489f96e5e2ca00352ce1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:38:19.169093 10186 raft_consensus.cc:504] T 00000000000000000000000000000000 P ffc098b71ea7489f96e5e2ca00352ce1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc098b71ea7489f96e5e2ca00352ce1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46195 } }
I20260619 14:38:19.169394 10186 leader_election.cc:302] T 00000000000000000000000000000000 P ffc098b71ea7489f96e5e2ca00352ce1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ffc098b71ea7489f96e5e2ca00352ce1; no voters:
I20260619 14:38:19.169673 10186 leader_election.cc:288] T 00000000000000000000000000000000 P ffc098b71ea7489f96e5e2ca00352ce1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:38:19.169745 10191 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ffc098b71ea7489f96e5e2ca00352ce1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:38:19.169953 10191 raft_consensus.cc:686] T 00000000000000000000000000000000 P ffc098b71ea7489f96e5e2ca00352ce1 [term 1 LEADER]: Becoming Leader. State: Replica: ffc098b71ea7489f96e5e2ca00352ce1, State: Running, Role: LEADER
I20260619 14:38:19.170336 10191 consensus_queue.cc:237] T 00000000000000000000000000000000 P ffc098b71ea7489f96e5e2ca00352ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc098b71ea7489f96e5e2ca00352ce1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46195 } }
I20260619 14:38:19.170786 10186 sys_catalog.cc:564] T 00000000000000000000000000000000 P ffc098b71ea7489f96e5e2ca00352ce1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:38:19.171941 10193 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffc098b71ea7489f96e5e2ca00352ce1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffc098b71ea7489f96e5e2ca00352ce1. Latest consensus state: current_term: 1 leader_uuid: "ffc098b71ea7489f96e5e2ca00352ce1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc098b71ea7489f96e5e2ca00352ce1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46195 } } }
I20260619 14:38:19.172062 10193 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffc098b71ea7489f96e5e2ca00352ce1 [sys.catalog]: This master's current role is: LEADER
I20260619 14:38:19.172407 10200 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:38:19.172370 10192 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffc098b71ea7489f96e5e2ca00352ce1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ffc098b71ea7489f96e5e2ca00352ce1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc098b71ea7489f96e5e2ca00352ce1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46195 } } }
I20260619 14:38:19.173117 10192 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffc098b71ea7489f96e5e2ca00352ce1 [sys.catalog]: This master's current role is: LEADER
I20260619 14:38:19.175326 10200 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:38:19.180480 10200 catalog_manager.cc:1347] Generated new cluster ID: 0a4d9e53ee854d68b0bdf6a3ce326f5d
I20260619 14:38:19.180538 10200 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:38:19.189517 10200 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:38:19.189810 10200 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:38:19.201870 10200 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ffc098b71ea7489f96e5e2ca00352ce1: Generated new TSK 0
I20260619 14:38:19.202539 10200 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:38:19.208598 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46195
--builtin_ntp_servers=127.0.98.20:35223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 14:38:19.331990 10210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:38:19.332249 10210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:38:19.332289 10210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:38:19.336508 10210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:38:19.336567 10210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:38:19.336624 10210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:38:19.341459 10210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:38:19.342479 10210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:38:19.343766 10210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:38:19.352844 10216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:38:19.352844 10218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:38:19.352854 10215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:38:19.353184 10210 server_base.cc:1034] running on GCE node
I20260619 14:38:19.353579 10210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:38:19.354182 10210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:38:19.355341 10210 hybrid_clock.cc:648] HybridClock initialized: now 1781879899355308 us; error 44 us; skew 500 ppm
I20260619 14:38:19.357081 10210 webserver.cc:460] Webserver started at http://127.0.98.1:39015/ using document root <none> and password file <none>
I20260619 14:38:19.357621 10210 fs_manager.cc:359] Metadata directory not provided
I20260619 14:38:19.357688 10210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:38:19.357874 10210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:38:19.359844 10210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/ts-0/data/instance:
uuid: "f82694118421494f9c7367eca770a045"
format_stamp: "Formatted at 2026-06-19 14:38:19 on dist-test-slave-32hj"
server_key: "f62e807e3ca50d0201c53704fe5350f7"
server_key_iv: "bad793e7eb18744c0733fb6238377c1f"
server_key_version: "encryptionkey@0"
I20260619 14:38:19.360378 10210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f82694118421494f9c7367eca770a045"
format_stamp: "Formatted at 2026-06-19 14:38:19 on dist-test-slave-32hj"
server_key: "f62e807e3ca50d0201c53704fe5350f7"
server_key_iv: "bad793e7eb18744c0733fb6238377c1f"
server_key_version: "encryptionkey@0"
I20260619 14:38:19.364343 10210 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 14:38:19.366645 10224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:38:19.367763 10210 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:38:19.367864 10210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/ts-0/wal
uuid: "f82694118421494f9c7367eca770a045"
format_stamp: "Formatted at 2026-06-19 14:38:19 on dist-test-slave-32hj"
server_key: "f62e807e3ca50d0201c53704fe5350f7"
server_key_iv: "bad793e7eb18744c0733fb6238377c1f"
server_key_version: "encryptionkey@0"
I20260619 14:38:19.367955 10210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:38:19.372361 10210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:38:19.372949 10210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:38:19.373095 10210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:38:19.373587 10210 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:38:19.374408 10210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:38:19.374451 10210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:38:19.374498 10210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:38:19.374526 10210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:38:19.382440 10210 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38145
I20260619 14:38:19.382476 10337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38145 every 8 connection(s)
I20260619 14:38:19.383255 10210 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:38:19.385243 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 10210
I20260619 14:38:19.385349 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:38:19.385596 392 external_mini_cluster.cc:1428] Setting key dc04aa54168f27282bef1d2ed4797add
I20260619 14:38:19.387683 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46195
--builtin_ntp_servers=127.0.98.20:35223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 14:38:19.389853 10189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:19.385051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55095 (local address 127.0.98.62:46195)
0619 14:38:19.385219 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:19.385225 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:19.386338 (+ 1113us) server_negotiation.cc:407] Connection header received
0619 14:38:19.387915 (+ 1577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:19.387918 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:19.387956 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:19.388007 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:19.389410 (+ 1403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:38:19.389414 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:38:19.389417 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:38:19.389431 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:38:19.389455 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:38:19.389467 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:38:19.389568 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 14:38:19.389670 (+ 102us) server_negotiation.cc:299] Negotiation successful
0619 14:38:19.389712 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 14:38:19.390599 10340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:19.385468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46195 (local address 127.0.98.1:55095)
0619 14:38:19.386174 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:19.386226 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:19.387582 (+ 1356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:19.388362 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:19.388371 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:19.388479 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:38:19.389118 (+ 639us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:38:19.389120 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:38:19.389179 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:38:19.389265 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:38:19.389553 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:38:19.389558 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:38:19.389562 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 14:38:19.389678 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 14:38:19.389872 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":124,"threads_started":1}
I20260619 14:38:19.391726 10338 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46195
I20260619 14:38:19.391986 10338 heartbeater.cc:464] Registering TS with master...
I20260619 14:38:19.392450 10338 heartbeater.cc:511] Master 127.0.98.62:46195 requested a full tablet report, sending...
I20260619 14:38:19.394047 10150 ts_manager.cc:194] Registered new tserver with Master: f82694118421494f9c7367eca770a045 (127.0.98.1:38145)
W20260619 14:38:19.499001 10341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:38:19.499282 10341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:38:19.499343 10341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:38:19.503348 10341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:38:19.503403 10341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:38:19.503489 10341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:38:19.508193 10341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:38:19.509327 10341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:38:19.510715 10341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:38:19.520376 10346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:38:19.520452 10341 server_base.cc:1034] running on GCE node
W20260619 14:38:19.520404 10347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:38:19.520376 10349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:38:19.521034 10341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:38:19.521642 10341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:38:19.522822 10341 hybrid_clock.cc:648] HybridClock initialized: now 1781879899522806 us; error 31 us; skew 500 ppm
I20260619 14:38:19.524712 10341 webserver.cc:460] Webserver started at http://127.0.98.2:34791/ using document root <none> and password file <none>
I20260619 14:38:19.525298 10341 fs_manager.cc:359] Metadata directory not provided
I20260619 14:38:19.525359 10341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:38:19.525550 10341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:38:19.527551 10341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/ts-1/data/instance:
uuid: "25d2c03192714e90a4b093e504dc0a72"
format_stamp: "Formatted at 2026-06-19 14:38:19 on dist-test-slave-32hj"
server_key: "36d061eef53c851739d227730c0d3adc"
server_key_iv: "7d44a3c7cd4ca726a2d91167c66725b8"
server_key_version: "encryptionkey@0"
I20260619 14:38:19.528070 10341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/ts-1/wal/instance:
uuid: "25d2c03192714e90a4b093e504dc0a72"
format_stamp: "Formatted at 2026-06-19 14:38:19 on dist-test-slave-32hj"
server_key: "36d061eef53c851739d227730c0d3adc"
server_key_iv: "7d44a3c7cd4ca726a2d91167c66725b8"
server_key_version: "encryptionkey@0"
I20260619 14:38:19.531705 10341 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 14:38:19.534245 10355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:38:19.535602 10341 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:38:19.535715 10341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/ts-1/wal
uuid: "25d2c03192714e90a4b093e504dc0a72"
format_stamp: "Formatted at 2026-06-19 14:38:19 on dist-test-slave-32hj"
server_key: "36d061eef53c851739d227730c0d3adc"
server_key_iv: "7d44a3c7cd4ca726a2d91167c66725b8"
server_key_version: "encryptionkey@0"
I20260619 14:38:19.535837 10341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:38:19.540398 10341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:38:19.541043 10341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:38:19.541199 10341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:38:19.541704 10341 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:38:19.542562 10341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:38:19.542604 10341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:38:19.542644 10341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:38:19.542672 10341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:38:19.550868 10341 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39527
I20260619 14:38:19.550891 10468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39527 every 8 connection(s)
I20260619 14:38:19.551703 10341 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:38:19.553921 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 10341
I20260619 14:38:19.554036 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:38:19.554359 392 external_mini_cluster.cc:1428] Setting key 1cfa4bc4df16af3d13f80d59262710f6
I20260619 14:38:19.556456 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46195
--builtin_ntp_servers=127.0.98.20:35223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 14:38:19.558058 10189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:19.553457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57605 (local address 127.0.98.62:46195)
0619 14:38:19.553644 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:19.553649 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:19.554819 (+ 1170us) server_negotiation.cc:407] Connection header received
0619 14:38:19.556256 (+ 1437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:19.556258 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:19.556290 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:19.556355 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:19.557584 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:38:19.557587 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:38:19.557589 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:38:19.557600 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:38:19.557620 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:38:19.557628 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:38:19.557719 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 14:38:19.557873 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 14:38:19.557930 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 14:38:19.558916 10471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:19.553891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46195 (local address 127.0.98.2:57605)
0619 14:38:19.554646 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:19.554700 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:19.556041 (+ 1341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:19.556533 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:19.556542 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:19.556659 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:38:19.557284 (+ 625us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:38:19.557286 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:38:19.557342 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:38:19.557426 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:38:19.557728 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:38:19.557737 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:38:19.557743 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 14:38:19.557904 (+ 161us) client_negotiation.cc:240] Negotiation successful
0619 14:38:19.558155 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":110,"threads_started":1}
I20260619 14:38:19.560045 10469 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46195
I20260619 14:38:19.560293 10469 heartbeater.cc:464] Registering TS with master...
I20260619 14:38:19.560674 10469 heartbeater.cc:511] Master 127.0.98.62:46195 requested a full tablet report, sending...
I20260619 14:38:19.561767 10150 ts_manager.cc:194] Registered new tserver with Master: 25d2c03192714e90a4b093e504dc0a72 (127.0.98.2:39527)
W20260619 14:38:19.665661 10472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:38:19.665933 10472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:38:19.665987 10472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:38:19.669979 10472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:38:19.670032 10472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:38:19.670104 10472 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:38:19.675329 10472 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:38:19.676386 10472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:38:19.677790 10472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:38:19.687049 10480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:38:19.687088 10477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:38:19.687227 10472 server_base.cc:1034] running on GCE node
W20260619 14:38:19.687259 10478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:38:19.687762 10472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:38:19.688350 10472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:38:19.689535 10472 hybrid_clock.cc:648] HybridClock initialized: now 1781879899689506 us; error 40 us; skew 500 ppm
I20260619 14:38:19.691394 10472 webserver.cc:460] Webserver started at http://127.0.98.3:44507/ using document root <none> and password file <none>
I20260619 14:38:19.691956 10472 fs_manager.cc:359] Metadata directory not provided
I20260619 14:38:19.692027 10472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:38:19.692214 10472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:38:19.694234 10472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/ts-2/data/instance:
uuid: "ddc9c329471b4aa999f99a6789fe04e3"
format_stamp: "Formatted at 2026-06-19 14:38:19 on dist-test-slave-32hj"
server_key: "4c64b1e2937b526c65e4454597a85191"
server_key_iv: "63fc96f98f34a7a823247c006f6445fb"
server_key_version: "encryptionkey@0"
I20260619 14:38:19.694754 10472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ddc9c329471b4aa999f99a6789fe04e3"
format_stamp: "Formatted at 2026-06-19 14:38:19 on dist-test-slave-32hj"
server_key: "4c64b1e2937b526c65e4454597a85191"
server_key_iv: "63fc96f98f34a7a823247c006f6445fb"
server_key_version: "encryptionkey@0"
I20260619 14:38:19.698460 10472 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260619 14:38:19.700802 10486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:38:19.702112 10472 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260619 14:38:19.702245 10472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/ts-2/wal
uuid: "ddc9c329471b4aa999f99a6789fe04e3"
format_stamp: "Formatted at 2026-06-19 14:38:19 on dist-test-slave-32hj"
server_key: "4c64b1e2937b526c65e4454597a85191"
server_key_iv: "63fc96f98f34a7a823247c006f6445fb"
server_key_version: "encryptionkey@0"
I20260619 14:38:19.702353 10472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:38:19.706800 10472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:38:19.707374 10472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:38:19.707525 10472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:38:19.708035 10472 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:38:19.708927 10472 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:38:19.708974 10472 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:38:19.709012 10472 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:38:19.709040 10472 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:38:19.717077 10472 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39335
I20260619 14:38:19.717085 10599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39335 every 8 connection(s)
I20260619 14:38:19.717957 10472 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:38:19.722671 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 10472
I20260619 14:38:19.722791 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781879837648036-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:38:19.723059 392 external_mini_cluster.cc:1428] Setting key 664e9bc8b95178464fce6f6fbd827bbb
I20260619 14:38:19.723927 10189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:19.719733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50567 (local address 127.0.98.62:46195)
0619 14:38:19.719895 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:19.719900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:19.720955 (+ 1055us) server_negotiation.cc:407] Connection header received
0619 14:38:19.722184 (+ 1229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:19.722186 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:19.722225 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:19.722301 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:19.723468 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:38:19.723471 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:38:19.723474 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:38:19.723487 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:38:19.723509 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:38:19.723519 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:38:19.723600 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 14:38:19.723754 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 14:38:19.723797 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 14:38:19.724951 10602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:19.720149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46195 (local address 127.0.98.3:50567)
0619 14:38:19.720815 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:19.720856 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:19.721992 (+ 1136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:19.722490 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:19.722499 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:19.722613 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:38:19.723170 (+ 557us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:38:19.723173 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:38:19.723225 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:38:19.723316 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:38:19.723608 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:38:19.723616 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:38:19.723621 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 14:38:19.723781 (+ 160us) client_negotiation.cc:240] Negotiation successful
0619 14:38:19.724029 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":104,"threads_started":1}
I20260619 14:38:19.725953 10600 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46195
I20260619 14:38:19.726159 10600 heartbeater.cc:464] Registering TS with master...
I20260619 14:38:19.726567 10600 heartbeater.cc:511] Master 127.0.98.62:46195 requested a full tablet report, sending...
I20260619 14:38:19.727568 10150 ts_manager.cc:194] Registered new tserver with Master: ddc9c329471b4aa999f99a6789fe04e3 (127.0.98.3:39335)
I20260619 14:38:19.736987 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 14:38:19.739862 10189 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:38:19.739040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58970 (local address 127.0.98.62:46195)
0619 14:38:19.739165 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:19.739169 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:19.739283 (+ 114us) server_negotiation.cc:407] Connection header received
0619 14:38:19.739393 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:19.739395 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:19.739429 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:19.739497 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:19.739731 (+ 234us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58970: BlockingRecv error: recv got EOF from 127.0.0.1:58970 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 14:38:19.740010 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 10210
I20260619 14:38:19.744820 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 10341
I20260619 14:38:19.749408 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 10472
I20260619 14:38:19.754182 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 10119
2026-06-19T14:38:19Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (785 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:38:19 dist-test-slave-32hj krb5kdc[10610](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:38:19 dist-test-slave-32hj krb5kdc[10610](info): set up 2 sockets
Jun 19 14:38:19 dist-test-slave-32hj krb5kdc[10610](info): commencing operation
krb5kdc: starting...
W20260619 14:38:20.713295 392 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/krb5kdc/test-user.keytab.
Jun 19 14:38:20 dist-test-slave-32hj krb5kdc[10610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:38:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:38:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:38:20.823812 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33919
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33919
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:38:20.929414 10623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:38:20.929677 10623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:38:20.929741 10623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:38:20.933780 10623 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:38:20.933844 10623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:38:20.933872 10623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:38:20.933893 10623 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:38:20.933919 10623 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:38:20.939134 10623 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33919
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:38:20.940168 10623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:38:20.941263 10623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:38:20.949043 10623 server_base.cc:1034] running on GCE node
W20260619 14:38:20.949080 10628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:38:20.949023 10629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:38:20.949067 10631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:38:20.949817 10623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:38:20.950836 10623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:38:20.951996 10623 hybrid_clock.cc:648] HybridClock initialized: now 1781879900951979 us; error 29 us; skew 500 ppm
Jun 19 14:38:20 dist-test-slave-32hj krb5kdc[10610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879900, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:38:20.954607 10623 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:38:20.955448 10623 webserver.cc:460] Webserver started at http://127.0.98.62:45265/ using document root <none> and password file <none>
I20260619 14:38:20.955966 10623 fs_manager.cc:359] Metadata directory not provided
I20260619 14:38:20.956028 10623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:38:20.956209 10623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:38:20.958164 10623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/master-0/data/instance:
uuid: "aa85dd2f985d4f8290c3bc9d7e0e4785"
format_stamp: "Formatted at 2026-06-19 14:38:20 on dist-test-slave-32hj"
server_key: "8e0f895062d2340a286107af48fe49c4"
server_key_iv: "b68ad2d7be1c7878e72a0dd9cefe4e49"
server_key_version: "encryptionkey@0"
I20260619 14:38:20.958674 10623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/master-0/wal/instance:
uuid: "aa85dd2f985d4f8290c3bc9d7e0e4785"
format_stamp: "Formatted at 2026-06-19 14:38:20 on dist-test-slave-32hj"
server_key: "8e0f895062d2340a286107af48fe49c4"
server_key_iv: "b68ad2d7be1c7878e72a0dd9cefe4e49"
server_key_version: "encryptionkey@0"
I20260619 14:38:20.962203 10623 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:38:20.964637 10638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:38:20.965992 10623 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:38:20.966110 10623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/master-0/wal
uuid: "aa85dd2f985d4f8290c3bc9d7e0e4785"
format_stamp: "Formatted at 2026-06-19 14:38:20 on dist-test-slave-32hj"
server_key: "8e0f895062d2340a286107af48fe49c4"
server_key_iv: "b68ad2d7be1c7878e72a0dd9cefe4e49"
server_key_version: "encryptionkey@0"
I20260619 14:38:20.966217 10623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:38:20.999750 10623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:38:21.003159 10623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:38:21.003355 10623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:38:21.010311 10623 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33919
I20260619 14:38:21.010663 10690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33919 every 8 connection(s)
I20260619 14:38:21.011304 10623 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:38:21.014057 10691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:38:21.019327 10691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa85dd2f985d4f8290c3bc9d7e0e4785: Bootstrap starting.
I20260619 14:38:21.019829 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 10623
I20260619 14:38:21.019951 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/master-0/wal/instance
I20260619 14:38:21.020218 392 external_mini_cluster.cc:1428] Setting key a425a37a48f81e20024b2d8562d463ee
I20260619 14:38:21.021996 10691 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa85dd2f985d4f8290c3bc9d7e0e4785: Neither blocks nor log segments found. Creating new log.
I20260619 14:38:21.022682 10691 log.cc:824] T 00000000000000000000000000000000 P aa85dd2f985d4f8290c3bc9d7e0e4785: Log is configured to *not* fsync() on all Append() calls
I20260619 14:38:21.024758 10691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa85dd2f985d4f8290c3bc9d7e0e4785: No bootstrap required, opened a new log
I20260619 14:38:21.028460 10691 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa85dd2f985d4f8290c3bc9d7e0e4785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa85dd2f985d4f8290c3bc9d7e0e4785" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33919 } }
I20260619 14:38:21.028661 10691 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa85dd2f985d4f8290c3bc9d7e0e4785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:38:21.028692 10691 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa85dd2f985d4f8290c3bc9d7e0e4785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa85dd2f985d4f8290c3bc9d7e0e4785, State: Initialized, Role: FOLLOWER
I20260619 14:38:21.029111 10691 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa85dd2f985d4f8290c3bc9d7e0e4785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa85dd2f985d4f8290c3bc9d7e0e4785" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33919 } }
I20260619 14:38:21.029225 10691 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa85dd2f985d4f8290c3bc9d7e0e4785 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:38:21.029290 10691 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa85dd2f985d4f8290c3bc9d7e0e4785 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:38:21.029379 10691 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa85dd2f985d4f8290c3bc9d7e0e4785 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:38:21.030161 10691 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa85dd2f985d4f8290c3bc9d7e0e4785 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa85dd2f985d4f8290c3bc9d7e0e4785" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33919 } }
I20260619 14:38:21.030471 10691 leader_election.cc:302] T 00000000000000000000000000000000 P aa85dd2f985d4f8290c3bc9d7e0e4785 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa85dd2f985d4f8290c3bc9d7e0e4785; no voters:
I20260619 14:38:21.030740 10691 leader_election.cc:288] T 00000000000000000000000000000000 P aa85dd2f985d4f8290c3bc9d7e0e4785 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:38:21.030844 10696 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa85dd2f985d4f8290c3bc9d7e0e4785 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:38:21.031062 10696 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa85dd2f985d4f8290c3bc9d7e0e4785 [term 1 LEADER]: Becoming Leader. State: Replica: aa85dd2f985d4f8290c3bc9d7e0e4785, State: Running, Role: LEADER
I20260619 14:38:21.031363 10696 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa85dd2f985d4f8290c3bc9d7e0e4785 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa85dd2f985d4f8290c3bc9d7e0e4785" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33919 } }
I20260619 14:38:21.032009 10691 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa85dd2f985d4f8290c3bc9d7e0e4785 [sys.catalog]: configured and running, proceeding with master startup.
Jun 19 14:38:21 dist-test-slave-32hj krb5kdc[10610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:38:21.032989 10697 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa85dd2f985d4f8290c3bc9d7e0e4785 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa85dd2f985d4f8290c3bc9d7e0e4785" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa85dd2f985d4f8290c3bc9d7e0e4785" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33919 } } }
I20260619 14:38:21.033119 10697 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa85dd2f985d4f8290c3bc9d7e0e4785 [sys.catalog]: This master's current role is: LEADER
I20260619 14:38:21.033383 10698 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa85dd2f985d4f8290c3bc9d7e0e4785 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa85dd2f985d4f8290c3bc9d7e0e4785. Latest consensus state: current_term: 1 leader_uuid: "aa85dd2f985d4f8290c3bc9d7e0e4785" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa85dd2f985d4f8290c3bc9d7e0e4785" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33919 } } }
I20260619 14:38:21.033473 10698 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa85dd2f985d4f8290c3bc9d7e0e4785 [sys.catalog]: This master's current role is: LEADER
I20260619 14:38:21.033807 10705 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:38:21.036878 10705 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:38:21.038349 10694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:21.021527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34454 (local address 127.0.98.62:33919)
0619 14:38:21.021911 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:21.021927 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:21.021963 (+ 36us) server_negotiation.cc:407] Connection header received
0619 14:38:21.022631 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:21.022649 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:21.023015 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:21.023392 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:21.024151 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:21.025229 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:21.026372 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:21.026851 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:21.026975 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:21.033082 (+ 6107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:38:21.033088 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:38:21.033105 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:38:21.033137 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:38:21.035190 (+ 2053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:38:21.035665 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:38:21.035671 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:38:21.035677 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:38:21.035810 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:38:21.036096 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:38:21.036100 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:38:21.036101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:38:21.036580 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:38:21.036810 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0619 14:38:21.037126 (+ 316us) server_negotiation.cc:299] Negotiation successful
0619 14:38:21.037434 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":103,"threads_started":1}
I20260619 14:38:21.042594 10705 catalog_manager.cc:1347] Generated new cluster ID: 4e9b669e9ba148af9321b980eca3c2ae
I20260619 14:38:21.042660 10705 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:38:21.050119 10705 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:38:21.051318 10705 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:38:21.058853 10705 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa85dd2f985d4f8290c3bc9d7e0e4785: Generated new TSK 0
I20260619 14:38:21.059667 10705 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 14:38:21.093951 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33919
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:38:21.206429 10717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:38:21.206686 10717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:38:21.206740 10717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:38:21.210692 10717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:38:21.210757 10717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:38:21.210835 10717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:38:21.216037 10717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:38:21.217097 10717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:38:21.218079 10717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:38:21.227118 10725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:38:21.227180 10722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:38:21.227262 10717 server_base.cc:1034] running on GCE node
W20260619 14:38:21.227088 10723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:38:21.227728 10717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:38:21.228330 10717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:38:21.229492 10717 hybrid_clock.cc:648] HybridClock initialized: now 1781879901229471 us; error 33 us; skew 500 ppm
Jun 19 14:38:21 dist-test-slave-32hj krb5kdc[10610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879901, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:38:21.232196 10717 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:38:21.233162 10717 webserver.cc:460] Webserver started at http://127.0.98.1:34731/ using document root <none> and password file <none>
I20260619 14:38:21.233724 10717 fs_manager.cc:359] Metadata directory not provided
I20260619 14:38:21.233789 10717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:38:21.233980 10717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:38:21.235982 10717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/ts-0/data/instance:
uuid: "a24847929dc7479095ffc1b0520bdc3c"
format_stamp: "Formatted at 2026-06-19 14:38:21 on dist-test-slave-32hj"
server_key: "3b523d97e0008fb1e4384bec739bb822"
server_key_iv: "cea9a0a2fa4b676eb758ac04ae595bec"
server_key_version: "encryptionkey@0"
I20260619 14:38:21.236558 10717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a24847929dc7479095ffc1b0520bdc3c"
format_stamp: "Formatted at 2026-06-19 14:38:21 on dist-test-slave-32hj"
server_key: "3b523d97e0008fb1e4384bec739bb822"
server_key_iv: "cea9a0a2fa4b676eb758ac04ae595bec"
server_key_version: "encryptionkey@0"
I20260619 14:38:21.240384 10717 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260619 14:38:21.242870 10732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:38:21.244190 10717 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:38:21.244311 10717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/ts-0/wal
uuid: "a24847929dc7479095ffc1b0520bdc3c"
format_stamp: "Formatted at 2026-06-19 14:38:21 on dist-test-slave-32hj"
server_key: "3b523d97e0008fb1e4384bec739bb822"
server_key_iv: "cea9a0a2fa4b676eb758ac04ae595bec"
server_key_version: "encryptionkey@0"
I20260619 14:38:21.244426 10717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:38:21.293875 10717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:38:21.297235 10717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:38:21.297425 10717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:38:21.298017 10717 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:38:21.298840 10717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:38:21.298887 10717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:38:21.298939 10717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:38:21.298967 10717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:38:21.307718 10717 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42881
I20260619 14:38:21.307746 10845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42881 every 8 connection(s)
I20260619 14:38:21.308605 10717 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:38:21.310891 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 10717
I20260619 14:38:21.310994 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:38:21.311262 392 external_mini_cluster.cc:1428] Setting key 117817bdca2aa59bce1261c659b19208
Jun 19 14:38:21 dist-test-slave-32hj krb5kdc[10610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879901, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 14:38:21.322726 10694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:21.310369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56779 (local address 127.0.98.62:33919)
0619 14:38:21.310502 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:21.310506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:21.311370 (+ 864us) server_negotiation.cc:407] Connection header received
0619 14:38:21.312873 (+ 1503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:21.312877 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:21.312927 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:21.312997 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:21.314178 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:21.314738 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:21.315945 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:21.316183 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:21.316287 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:21.319074 (+ 2787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:38:21.319078 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:38:21.319080 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:38:21.319105 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:38:21.320515 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:38:21.321105 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:38:21.321109 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:38:21.321111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:38:21.321191 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:38:21.321553 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:38:21.321555 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:38:21.321557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:38:21.321739 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:38:21.321827 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 14:38:21.322419 (+ 592us) server_negotiation.cc:299] Negotiation successful
0619 14:38:21.322552 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 14:38:21.323539 10848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:21.310743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33919 (local address 127.0.98.1:56779)
0619 14:38:21.311210 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:21.311269 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:21.312641 (+ 1372us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:21.313163 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:21.313171 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:21.313519 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:38:21.314041 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:21.314060 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:21.314867 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:21.314871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:21.315821 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:21.315830 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:21.316341 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:21.316345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:21.316507 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:21.317078 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:38:21.317099 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:38:21.318872 (+ 1773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:38:21.320632 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:38:21.320637 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:38:21.320651 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:38:21.321005 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:38:21.321284 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:38:21.321287 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:38:21.321289 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:38:21.321443 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:38:21.321866 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:38:21.321873 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:38:21.322163 (+ 290us) client_negotiation.cc:769] Sending connection context
0619 14:38:21.322374 (+ 211us) client_negotiation.cc:240] Negotiation successful
0619 14:38:21.322721 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":113,"threads_started":1}
I20260619 14:38:21.325038 10846 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33919
I20260619 14:38:21.325316 10846 heartbeater.cc:464] Registering TS with master...
I20260619 14:38:21.325868 10846 heartbeater.cc:511] Master 127.0.98.62:33919 requested a full tablet report, sending...
I20260619 14:38:21.327450 10655 ts_manager.cc:194] Registered new tserver with Master: a24847929dc7479095ffc1b0520bdc3c (127.0.98.1:42881)
I20260619 14:38:21.328557 10655 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56779
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 14:38:21.340075 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33919
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:38:21.448091 10851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:38:21.448369 10851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:38:21.448427 10851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:38:21.452507 10851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:38:21.452584 10851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:38:21.452643 10851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:38:21.458020 10851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:38:21.459061 10851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:38:21.460026 10851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:38:21.469543 10856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:38:21.469587 10851 server_base.cc:1034] running on GCE node
W20260619 14:38:21.469579 10857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:38:21.469594 10859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:38:21.470306 10851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:38:21.470990 10851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:38:21.472149 10851 hybrid_clock.cc:648] HybridClock initialized: now 1781879901472130 us; error 35 us; skew 500 ppm
Jun 19 14:38:21 dist-test-slave-32hj krb5kdc[10610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879901, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:38:21.474874 10851 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:38:21.475791 10851 webserver.cc:460] Webserver started at http://127.0.98.2:36705/ using document root <none> and password file <none>
I20260619 14:38:21.476346 10851 fs_manager.cc:359] Metadata directory not provided
I20260619 14:38:21.476413 10851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:38:21.476600 10851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:38:21.478693 10851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/ts-1/data/instance:
uuid: "1cb09abcfb6d42a4a841e6c77b8fb18a"
format_stamp: "Formatted at 2026-06-19 14:38:21 on dist-test-slave-32hj"
server_key: "bc750000cb48a537bdc1968015f49331"
server_key_iv: "1154fc22ff299428a64b4c6c90096a73"
server_key_version: "encryptionkey@0"
I20260619 14:38:21.479254 10851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1cb09abcfb6d42a4a841e6c77b8fb18a"
format_stamp: "Formatted at 2026-06-19 14:38:21 on dist-test-slave-32hj"
server_key: "bc750000cb48a537bdc1968015f49331"
server_key_iv: "1154fc22ff299428a64b4c6c90096a73"
server_key_version: "encryptionkey@0"
I20260619 14:38:21.483184 10851 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:38:21.485905 10866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:38:21.487329 10851 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:38:21.487462 10851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/ts-1/wal
uuid: "1cb09abcfb6d42a4a841e6c77b8fb18a"
format_stamp: "Formatted at 2026-06-19 14:38:21 on dist-test-slave-32hj"
server_key: "bc750000cb48a537bdc1968015f49331"
server_key_iv: "1154fc22ff299428a64b4c6c90096a73"
server_key_version: "encryptionkey@0"
I20260619 14:38:21.487584 10851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:38:21.505311 10851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:38:21.508517 10851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:38:21.508692 10851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:38:21.509322 10851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:38:21.510134 10851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:38:21.510178 10851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:38:21.510228 10851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:38:21.510257 10851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:38:21.518384 10851 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42967
I20260619 14:38:21.518410 10979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42967 every 8 connection(s)
I20260619 14:38:21.519261 10851 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:38:21.526875 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 10851
I20260619 14:38:21.527002 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:38:21.527340 392 external_mini_cluster.cc:1428] Setting key 965f2a2ae1628f1d97ebbcaa3fdeb91b
Jun 19 14:38:21 dist-test-slave-32hj krb5kdc[10610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879901, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:38:21.533428 10694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:21.521037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42917 (local address 127.0.98.62:33919)
0619 14:38:21.521181 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:21.521186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:21.522064 (+ 878us) server_negotiation.cc:407] Connection header received
0619 14:38:21.523303 (+ 1239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:21.523306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:21.523355 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:21.523432 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:21.524648 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:21.525253 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:21.526435 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:21.526649 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:21.526728 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:21.529593 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:38:21.529596 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:38:21.529598 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:38:21.529621 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:38:21.531151 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:38:21.531669 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:38:21.531673 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:38:21.531675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:38:21.531764 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:38:21.532132 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:38:21.532141 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:38:21.532143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:38:21.532343 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:38:21.532489 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 14:38:21.533115 (+ 626us) server_negotiation.cc:299] Negotiation successful
0619 14:38:21.533266 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 14:38:21.534269 10982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:21.521456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33919 (local address 127.0.98.2:42917)
0619 14:38:21.521900 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:21.521942 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:21.523074 (+ 1132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:21.523624 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:21.523633 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:21.523990 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:38:21.524511 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:21.524524 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:21.525391 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:21.525394 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:21.526318 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:21.526327 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:21.526836 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:21.526840 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:21.527046 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:21.527650 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:38:21.527670 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:38:21.529414 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:38:21.531285 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:38:21.531290 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:38:21.531301 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:38:21.531551 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:38:21.531901 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:38:21.531903 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:38:21.531905 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:38:21.532043 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:38:21.532469 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:38:21.532475 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:38:21.532868 (+ 393us) client_negotiation.cc:769] Sending connection context
0619 14:38:21.533039 (+ 171us) client_negotiation.cc:240] Negotiation successful
0619 14:38:21.533323 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":116,"threads_started":1}
I20260619 14:38:21.535521 10980 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33919
I20260619 14:38:21.535777 10980 heartbeater.cc:464] Registering TS with master...
I20260619 14:38:21.536268 10980 heartbeater.cc:511] Master 127.0.98.62:33919 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 14:38:21.538244 10655 ts_manager.cc:194] Registered new tserver with Master: 1cb09abcfb6d42a4a841e6c77b8fb18a (127.0.98.2:42967)
I20260619 14:38:21.538790 10655 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42917
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 14:38:21.556798 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33919
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:38:21.663298 10985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:38:21.663563 10985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:38:21.663621 10985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:38:21.667834 10985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:38:21.667899 10985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:38:21.667980 10985 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:38:21.673413 10985 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:38:21.674422 10985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:38:21.675407 10985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:38:21.684964 10990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:38:21.685017 10985 server_base.cc:1034] running on GCE node
W20260619 14:38:21.684968 10991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:38:21.684964 10993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:38:21.685585 10985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:38:21.686218 10985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:38:21.687384 10985 hybrid_clock.cc:648] HybridClock initialized: now 1781879901687366 us; error 28 us; skew 500 ppm
Jun 19 14:38:21 dist-test-slave-32hj krb5kdc[10610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879901, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:38:21.689982 10985 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:38:21.690904 10985 webserver.cc:460] Webserver started at http://127.0.98.3:34871/ using document root <none> and password file <none>
I20260619 14:38:21.691431 10985 fs_manager.cc:359] Metadata directory not provided
I20260619 14:38:21.691493 10985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:38:21.691679 10985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:38:21.693769 10985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/ts-2/data/instance:
uuid: "e5f6c1ab1def4977b2ef5a61ae7c7b0a"
format_stamp: "Formatted at 2026-06-19 14:38:21 on dist-test-slave-32hj"
server_key: "fc39cdc5d1db9bec09893db19ca8429e"
server_key_iv: "8259435f2d41e3320e5c6a2e7cf6f57f"
server_key_version: "encryptionkey@0"
I20260619 14:38:21.694278 10985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e5f6c1ab1def4977b2ef5a61ae7c7b0a"
format_stamp: "Formatted at 2026-06-19 14:38:21 on dist-test-slave-32hj"
server_key: "fc39cdc5d1db9bec09893db19ca8429e"
server_key_iv: "8259435f2d41e3320e5c6a2e7cf6f57f"
server_key_version: "encryptionkey@0"
I20260619 14:38:21.697746 10985 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 14:38:21.699879 11000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:38:21.701052 10985 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:38:21.701143 10985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/ts-2/wal
uuid: "e5f6c1ab1def4977b2ef5a61ae7c7b0a"
format_stamp: "Formatted at 2026-06-19 14:38:21 on dist-test-slave-32hj"
server_key: "fc39cdc5d1db9bec09893db19ca8429e"
server_key_iv: "8259435f2d41e3320e5c6a2e7cf6f57f"
server_key_version: "encryptionkey@0"
I20260619 14:38:21.701238 10985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:38:21.714844 10985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:38:21.717760 10985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:38:21.717914 10985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:38:21.718487 10985 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:38:21.719267 10985 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:38:21.719313 10985 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:38:21.719350 10985 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:38:21.719364 10985 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:38:21.727604 10985 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46113
I20260619 14:38:21.727689 11113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46113 every 8 connection(s)
I20260619 14:38:21.728471 10985 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:38:21.733002 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 10985
I20260619 14:38:21.733155 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:38:21.733439 392 external_mini_cluster.cc:1428] Setting key d613e7effbf1b1c623a3179bb68268b4
Jun 19 14:38:21 dist-test-slave-32hj krb5kdc[10610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879901, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:38:21.742018 10694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:21.730301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40017 (local address 127.0.98.62:33919)
0619 14:38:21.730450 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:21.730454 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:21.731266 (+ 812us) server_negotiation.cc:407] Connection header received
0619 14:38:21.732646 (+ 1380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:21.732650 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:21.732706 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:21.732812 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:21.734209 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:21.734728 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:21.735852 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:21.736061 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:21.736146 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:21.738566 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:38:21.738569 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:38:21.738571 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:38:21.738595 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:38:21.739921 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:38:21.740427 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:38:21.740430 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:38:21.740431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:38:21.740514 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:38:21.740994 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:38:21.740997 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:38:21.740998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:38:21.741219 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:38:21.741304 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 14:38:21.741798 (+ 494us) server_negotiation.cc:299] Negotiation successful
0619 14:38:21.741897 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 14:38:21.742823 11116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:21.730649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33919 (local address 127.0.98.3:40017)
0619 14:38:21.731098 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:21.731141 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:21.732424 (+ 1283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:21.732986 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:21.732993 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:21.733451 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:38:21.734054 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:21.734070 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:21.734852 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:21.734855 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:21.735726 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:21.735735 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:21.736199 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:21.736203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:21.736365 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:21.736981 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:38:21.737000 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:38:21.738415 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:38:21.740044 (+ 1629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:38:21.740071 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:38:21.740083 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:38:21.740334 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:38:21.740648 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:38:21.740651 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:38:21.740652 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:38:21.740879 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:38:21.741314 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:38:21.741319 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:38:21.741551 (+ 232us) client_negotiation.cc:769] Sending connection context
0619 14:38:21.741724 (+ 173us) client_negotiation.cc:240] Negotiation successful
0619 14:38:21.741977 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":113,"threads_started":1}
I20260619 14:38:21.743894 11114 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33919
I20260619 14:38:21.744103 11114 heartbeater.cc:464] Registering TS with master...
I20260619 14:38:21.744526 11114 heartbeater.cc:511] Master 127.0.98.62:33919 requested a full tablet report, sending...
I20260619 14:38:21.745540 10655 ts_manager.cc:194] Registered new tserver with Master: e5f6c1ab1def4977b2ef5a61ae7c7b0a (127.0.98.3:46113)
I20260619 14:38:21.746202 10655 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40017
I20260619 14:38:21.747148 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:38:21.756047 10694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:21.749161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34458 (local address 127.0.98.62:33919)
0619 14:38:21.749328 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:21.749332 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:21.749477 (+ 145us) server_negotiation.cc:407] Connection header received
0619 14:38:21.749595 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:21.749598 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:21.749639 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:21.749701 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:38:21.750382 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:21.750883 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:21.751705 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:21.751932 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:21.752016 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:21.753273 (+ 1257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:38:21.753277 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:38:21.753280 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:38:21.753304 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:38:21.754465 (+ 1161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:38:21.754876 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:38:21.754879 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:38:21.754881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:38:21.754953 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:38:21.755194 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:38:21.755197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:38:21.755199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:38:21.755392 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:38:21.755502 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 14:38:21.755787 (+ 285us) server_negotiation.cc:299] Negotiation successful
0619 14:38:21.755884 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 14:38:21.758728 10655 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34458:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:38:21.761620 10655 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:38:21.775705 11127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:21.769389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42881 (local address 127.0.0.1:55360)
0619 14:38:21.770174 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:21.770222 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:21.770478 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:21.771204 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:21.771210 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:21.771232 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:38:21.771387 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:21.771394 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:21.772898 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:21.772901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:21.774217 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:21.774227 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:21.775195 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:21.775199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:21.775300 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:21.775340 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 14:38:21.775439 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 14:38:21.775555 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"mutex_wait_us":1,"thread_start_us":69,"threads_started":1}
I20260619 14:38:21.775732 11126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:21.769520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55360 (local address 127.0.98.1:42881)
0619 14:38:21.769917 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:21.769932 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:21.770367 (+ 435us) server_negotiation.cc:407] Connection header received
0619 14:38:21.770669 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:21.770675 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:21.770889 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:21.771036 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:38:21.771510 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:21.772767 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:21.774316 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:21.775073 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:21.775180 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:21.775322 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 14:38:21.775485 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 14:38:21.775582 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":86,"threads_started":1}
I20260619 14:38:21.775848 11130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:21.770351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43156 (local address 127.0.98.2:42967)
0619 14:38:21.770723 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:21.770730 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:21.771119 (+ 389us) server_negotiation.cc:407] Connection header received
0619 14:38:21.771197 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:21.771201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:21.771347 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:21.771482 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:38:21.771812 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:21.772587 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:21.774750 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:21.775336 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:21.775439 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:21.775562 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 14:38:21.775639 (+ 77us) server_negotiation.cc:299] Negotiation successful
0619 14:38:21.775721 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":139,"threads_started":1}
I20260619 14:38:21.776036 11125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:21.770127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42967 (local address 127.0.0.1:43156)
0619 14:38:21.771017 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:21.771029 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:21.771101 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:21.771619 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:21.771621 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:21.771635 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:38:21.771691 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:21.771694 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:21.772705 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:21.772708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:21.774634 (+ 1926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:21.774642 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:21.775441 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:21.775444 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:21.775528 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:21.775551 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 14:38:21.775603 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 14:38:21.775667 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":835,"mutex_wait_us":852,"thread_start_us":86,"threads_started":1}
I20260619 14:38:21.776480 11129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:21.769515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46113 (local address 127.0.0.1:58516)
0619 14:38:21.770815 (+ 1300us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:21.770825 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:21.770935 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:21.771353 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:21.771356 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:21.771371 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:38:21.771479 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:21.771485 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:21.773052 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:21.773055 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:21.774129 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:21.774138 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:21.776105 (+ 1967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:21.776109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:21.776212 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:21.776233 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 14:38:21.776302 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 14:38:21.776363 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1253,"mutex_wait_us":40,"thread_start_us":98,"threads_started":1}
I20260619 14:38:21.776795 11128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:21.769863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58516 (local address 127.0.98.3:46113)
0619 14:38:21.770580 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:21.770591 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:21.770944 (+ 353us) server_negotiation.cc:407] Connection header received
0619 14:38:21.771048 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:21.771053 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:21.771224 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:21.771367 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:38:21.771572 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:21.772772 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:21.774251 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:21.775073 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:21.775180 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:21.775322 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 14:38:21.776335 (+ 1013us) server_negotiation.cc:299] Negotiation successful
0619 14:38:21.776422 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":117,"threads_started":1}
I20260619 14:38:21.778699 10780 tablet_service.cc:1467] Processing CreateTablet for tablet 1236a038d8e8427da68078b863ac38bf (DEFAULT_TABLE table=test-table [id=5c7713cfb6154297b8e074e61ec94871]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:38:21.778812 10914 tablet_service.cc:1467] Processing CreateTablet for tablet 1236a038d8e8427da68078b863ac38bf (DEFAULT_TABLE table=test-table [id=5c7713cfb6154297b8e074e61ec94871]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:38:21.779637 10780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1236a038d8e8427da68078b863ac38bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:38:21.779383 11048 tablet_service.cc:1467] Processing CreateTablet for tablet 1236a038d8e8427da68078b863ac38bf (DEFAULT_TABLE table=test-table [id=5c7713cfb6154297b8e074e61ec94871]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:38:21.779815 10914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1236a038d8e8427da68078b863ac38bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:38:21.780278 11048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1236a038d8e8427da68078b863ac38bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:38:21.785198 11131 tablet_bootstrap.cc:492] T 1236a038d8e8427da68078b863ac38bf P a24847929dc7479095ffc1b0520bdc3c: Bootstrap starting.
I20260619 14:38:21.786598 11132 tablet_bootstrap.cc:492] T 1236a038d8e8427da68078b863ac38bf P e5f6c1ab1def4977b2ef5a61ae7c7b0a: Bootstrap starting.
I20260619 14:38:21.787765 11131 tablet_bootstrap.cc:654] T 1236a038d8e8427da68078b863ac38bf P a24847929dc7479095ffc1b0520bdc3c: Neither blocks nor log segments found. Creating new log.
I20260619 14:38:21.788659 11132 tablet_bootstrap.cc:654] T 1236a038d8e8427da68078b863ac38bf P e5f6c1ab1def4977b2ef5a61ae7c7b0a: Neither blocks nor log segments found. Creating new log.
I20260619 14:38:21.789047 11131 log.cc:824] T 1236a038d8e8427da68078b863ac38bf P a24847929dc7479095ffc1b0520bdc3c: Log is configured to *not* fsync() on all Append() calls
I20260619 14:38:21.789783 11133 tablet_bootstrap.cc:492] T 1236a038d8e8427da68078b863ac38bf P 1cb09abcfb6d42a4a841e6c77b8fb18a: Bootstrap starting.
I20260619 14:38:21.789841 11132 log.cc:824] T 1236a038d8e8427da68078b863ac38bf P e5f6c1ab1def4977b2ef5a61ae7c7b0a: Log is configured to *not* fsync() on all Append() calls
I20260619 14:38:21.790990 11131 tablet_bootstrap.cc:492] T 1236a038d8e8427da68078b863ac38bf P a24847929dc7479095ffc1b0520bdc3c: No bootstrap required, opened a new log
I20260619 14:38:21.791177 11131 ts_tablet_manager.cc:1397] T 1236a038d8e8427da68078b863ac38bf P a24847929dc7479095ffc1b0520bdc3c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 14:38:21.791821 11132 tablet_bootstrap.cc:492] T 1236a038d8e8427da68078b863ac38bf P e5f6c1ab1def4977b2ef5a61ae7c7b0a: No bootstrap required, opened a new log
I20260619 14:38:21.792003 11132 ts_tablet_manager.cc:1397] T 1236a038d8e8427da68078b863ac38bf P e5f6c1ab1def4977b2ef5a61ae7c7b0a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 14:38:21.792112 11133 tablet_bootstrap.cc:654] T 1236a038d8e8427da68078b863ac38bf P 1cb09abcfb6d42a4a841e6c77b8fb18a: Neither blocks nor log segments found. Creating new log.
I20260619 14:38:21.792902 11133 log.cc:824] T 1236a038d8e8427da68078b863ac38bf P 1cb09abcfb6d42a4a841e6c77b8fb18a: Log is configured to *not* fsync() on all Append() calls
I20260619 14:38:21.794581 11131 raft_consensus.cc:348] T 1236a038d8e8427da68078b863ac38bf P a24847929dc7479095ffc1b0520bdc3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a24847929dc7479095ffc1b0520bdc3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } } peers { permanent_uuid: "e5f6c1ab1def4977b2ef5a61ae7c7b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46113 } } peers { permanent_uuid: "1cb09abcfb6d42a4a841e6c77b8fb18a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42967 } }
I20260619 14:38:21.794801 11131 raft_consensus.cc:374] T 1236a038d8e8427da68078b863ac38bf P a24847929dc7479095ffc1b0520bdc3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:38:21.794860 11131 raft_consensus.cc:729] T 1236a038d8e8427da68078b863ac38bf P a24847929dc7479095ffc1b0520bdc3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a24847929dc7479095ffc1b0520bdc3c, State: Initialized, Role: FOLLOWER
I20260619 14:38:21.794884 11133 tablet_bootstrap.cc:492] T 1236a038d8e8427da68078b863ac38bf P 1cb09abcfb6d42a4a841e6c77b8fb18a: No bootstrap required, opened a new log
I20260619 14:38:21.795054 11133 ts_tablet_manager.cc:1397] T 1236a038d8e8427da68078b863ac38bf P 1cb09abcfb6d42a4a841e6c77b8fb18a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:38:21.795293 11131 consensus_queue.cc:260] T 1236a038d8e8427da68078b863ac38bf P a24847929dc7479095ffc1b0520bdc3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a24847929dc7479095ffc1b0520bdc3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } } peers { permanent_uuid: "e5f6c1ab1def4977b2ef5a61ae7c7b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46113 } } peers { permanent_uuid: "1cb09abcfb6d42a4a841e6c77b8fb18a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42967 } }
I20260619 14:38:21.795564 11132 raft_consensus.cc:348] T 1236a038d8e8427da68078b863ac38bf P e5f6c1ab1def4977b2ef5a61ae7c7b0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a24847929dc7479095ffc1b0520bdc3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } } peers { permanent_uuid: "e5f6c1ab1def4977b2ef5a61ae7c7b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46113 } } peers { permanent_uuid: "1cb09abcfb6d42a4a841e6c77b8fb18a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42967 } }
I20260619 14:38:21.795832 11132 raft_consensus.cc:374] T 1236a038d8e8427da68078b863ac38bf P e5f6c1ab1def4977b2ef5a61ae7c7b0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:38:21.795890 11132 raft_consensus.cc:729] T 1236a038d8e8427da68078b863ac38bf P e5f6c1ab1def4977b2ef5a61ae7c7b0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5f6c1ab1def4977b2ef5a61ae7c7b0a, State: Initialized, Role: FOLLOWER
I20260619 14:38:21.795923 10846 heartbeater.cc:503] Master 127.0.98.62:33919 was elected leader, sending a full tablet report...
I20260619 14:38:21.796324 11132 consensus_queue.cc:260] T 1236a038d8e8427da68078b863ac38bf P e5f6c1ab1def4977b2ef5a61ae7c7b0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a24847929dc7479095ffc1b0520bdc3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } } peers { permanent_uuid: "e5f6c1ab1def4977b2ef5a61ae7c7b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46113 } } peers { permanent_uuid: "1cb09abcfb6d42a4a841e6c77b8fb18a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42967 } }
I20260619 14:38:21.796967 11114 heartbeater.cc:503] Master 127.0.98.62:33919 was elected leader, sending a full tablet report...
I20260619 14:38:21.798079 11131 ts_tablet_manager.cc:1428] T 1236a038d8e8427da68078b863ac38bf P a24847929dc7479095ffc1b0520bdc3c: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260619 14:38:21.798671 11133 raft_consensus.cc:348] T 1236a038d8e8427da68078b863ac38bf P 1cb09abcfb6d42a4a841e6c77b8fb18a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a24847929dc7479095ffc1b0520bdc3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } } peers { permanent_uuid: "e5f6c1ab1def4977b2ef5a61ae7c7b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46113 } } peers { permanent_uuid: "1cb09abcfb6d42a4a841e6c77b8fb18a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42967 } }
I20260619 14:38:21.798923 11133 raft_consensus.cc:374] T 1236a038d8e8427da68078b863ac38bf P 1cb09abcfb6d42a4a841e6c77b8fb18a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:38:21.798978 11133 raft_consensus.cc:729] T 1236a038d8e8427da68078b863ac38bf P 1cb09abcfb6d42a4a841e6c77b8fb18a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb09abcfb6d42a4a841e6c77b8fb18a, State: Initialized, Role: FOLLOWER
I20260619 14:38:21.799388 11132 ts_tablet_manager.cc:1428] T 1236a038d8e8427da68078b863ac38bf P e5f6c1ab1def4977b2ef5a61ae7c7b0a: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260619 14:38:21.799418 11133 consensus_queue.cc:260] T 1236a038d8e8427da68078b863ac38bf P 1cb09abcfb6d42a4a841e6c77b8fb18a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a24847929dc7479095ffc1b0520bdc3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } } peers { permanent_uuid: "e5f6c1ab1def4977b2ef5a61ae7c7b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46113 } } peers { permanent_uuid: "1cb09abcfb6d42a4a841e6c77b8fb18a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42967 } }
I20260619 14:38:21.800242 11133 ts_tablet_manager.cc:1428] T 1236a038d8e8427da68078b863ac38bf P 1cb09abcfb6d42a4a841e6c77b8fb18a: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 14:38:21.800542 10980 heartbeater.cc:503] Master 127.0.98.62:33919 was elected leader, sending a full tablet report...
W20260619 14:38:21.809679 10847 tablet.cc:2369] T 1236a038d8e8427da68078b863ac38bf P a24847929dc7479095ffc1b0520bdc3c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:38:21.846612 11137 raft_consensus.cc:482] T 1236a038d8e8427da68078b863ac38bf P a24847929dc7479095ffc1b0520bdc3c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:38:21.846812 11137 raft_consensus.cc:504] T 1236a038d8e8427da68078b863ac38bf P a24847929dc7479095ffc1b0520bdc3c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a24847929dc7479095ffc1b0520bdc3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } } peers { permanent_uuid: "e5f6c1ab1def4977b2ef5a61ae7c7b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46113 } } peers { permanent_uuid: "1cb09abcfb6d42a4a841e6c77b8fb18a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42967 } }
I20260619 14:38:21.848034 11137 leader_election.cc:288] T 1236a038d8e8427da68078b863ac38bf P a24847929dc7479095ffc1b0520bdc3c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5f6c1ab1def4977b2ef5a61ae7c7b0a (127.0.98.3:46113), 1cb09abcfb6d42a4a841e6c77b8fb18a (127.0.98.2:42967)
I20260619 14:38:21.853147 11140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:21.848223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46113 (local address 127.0.98.1:44391)
0619 14:38:21.848553 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:21.848567 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:21.848685 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:21.849029 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:21.849033 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:21.849053 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:38:21.849202 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:21.849209 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:21.850207 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:21.850210 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:21.851684 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:21.851695 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:21.852548 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:21.852552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:21.852739 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:21.852828 (+ 89us) client_negotiation.cc:769] Sending connection context
0619 14:38:21.852909 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 14:38:21.852986 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":238,"spinlock_wait_cycles":11136,"thread_start_us":72,"threads_started":1}
I20260619 14:38:21.853164 11128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:21.848362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44391 (local address 127.0.98.3:46113)
0619 14:38:21.848491 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:21.848495 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:21.848653 (+ 158us) server_negotiation.cc:407] Connection header received
0619 14:38:21.848862 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:21.848865 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:21.848924 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:21.849008 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:38:21.849322 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:21.850105 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:21.851820 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:21.852435 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:21.852537 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:21.852599 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 14:38:21.852950 (+ 351us) server_negotiation.cc:299] Negotiation successful
0619 14:38:21.853044 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 14:38:21.853188 11130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:21.848395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54731 (local address 127.0.98.2:42967)
0619 14:38:21.848533 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:21.848537 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:21.848749 (+ 212us) server_negotiation.cc:407] Connection header received
0619 14:38:21.848926 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:21.848929 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:21.848976 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:21.849043 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:38:21.849311 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:21.850105 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:21.851820 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:21.852435 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:21.852536 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:21.852609 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 14:38:21.853016 (+ 407us) server_negotiation.cc:299] Negotiation successful
0619 14:38:21.853089 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 14:38:21.853616 11141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:21.848298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42967 (local address 127.0.98.1:54731)
0619 14:38:21.848650 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:21.848661 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:38:21.848818 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:38:21.849061 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:38:21.849063 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:38:21.849098 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:38:21.849209 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:21.849215 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:21.850206 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:21.850210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:21.851685 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:38:21.851695 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:38:21.852548 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:38:21.852552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:38:21.852739 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:21.852893 (+ 154us) client_negotiation.cc:769] Sending connection context
0619 14:38:21.853415 (+ 522us) client_negotiation.cc:240] Negotiation successful
0619 14:38:21.853485 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":218,"spinlock_wait_cycles":2176,"thread_start_us":116,"threads_started":1}
I20260619 14:38:21.853833 11068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1236a038d8e8427da68078b863ac38bf" candidate_uuid: "a24847929dc7479095ffc1b0520bdc3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5f6c1ab1def4977b2ef5a61ae7c7b0a" is_pre_election: true
I20260619 14:38:21.854157 11068 raft_consensus.cc:2454] T 1236a038d8e8427da68078b863ac38bf P e5f6c1ab1def4977b2ef5a61ae7c7b0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a24847929dc7479095ffc1b0520bdc3c in term 0.
I20260619 14:38:21.854239 10934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1236a038d8e8427da68078b863ac38bf" candidate_uuid: "a24847929dc7479095ffc1b0520bdc3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb09abcfb6d42a4a841e6c77b8fb18a" is_pre_election: true
I20260619 14:38:21.854525 10934 raft_consensus.cc:2454] T 1236a038d8e8427da68078b863ac38bf P 1cb09abcfb6d42a4a841e6c77b8fb18a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a24847929dc7479095ffc1b0520bdc3c in term 0.
I20260619 14:38:21.854629 10733 leader_election.cc:302] T 1236a038d8e8427da68078b863ac38bf P a24847929dc7479095ffc1b0520bdc3c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a24847929dc7479095ffc1b0520bdc3c, e5f6c1ab1def4977b2ef5a61ae7c7b0a; no voters:
I20260619 14:38:21.854905 11137 raft_consensus.cc:2781] T 1236a038d8e8427da68078b863ac38bf P a24847929dc7479095ffc1b0520bdc3c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:38:21.854981 11137 raft_consensus.cc:482] T 1236a038d8e8427da68078b863ac38bf P a24847929dc7479095ffc1b0520bdc3c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:38:21.855024 11137 raft_consensus.cc:3037] T 1236a038d8e8427da68078b863ac38bf P a24847929dc7479095ffc1b0520bdc3c [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:38:21.855991 11137 raft_consensus.cc:504] T 1236a038d8e8427da68078b863ac38bf P a24847929dc7479095ffc1b0520bdc3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a24847929dc7479095ffc1b0520bdc3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } } peers { permanent_uuid: "e5f6c1ab1def4977b2ef5a61ae7c7b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46113 } } peers { permanent_uuid: "1cb09abcfb6d42a4a841e6c77b8fb18a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42967 } }
I20260619 14:38:21.856307 11137 leader_election.cc:288] T 1236a038d8e8427da68078b863ac38bf P a24847929dc7479095ffc1b0520bdc3c [CANDIDATE]: Term 1 election: Requested vote from peers e5f6c1ab1def4977b2ef5a61ae7c7b0a (127.0.98.3:46113), 1cb09abcfb6d42a4a841e6c77b8fb18a (127.0.98.2:42967)
I20260619 14:38:21.856612 11068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1236a038d8e8427da68078b863ac38bf" candidate_uuid: "a24847929dc7479095ffc1b0520bdc3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5f6c1ab1def4977b2ef5a61ae7c7b0a"
I20260619 14:38:21.856639 10934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1236a038d8e8427da68078b863ac38bf" candidate_uuid: "a24847929dc7479095ffc1b0520bdc3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb09abcfb6d42a4a841e6c77b8fb18a"
I20260619 14:38:21.856711 11068 raft_consensus.cc:3037] T 1236a038d8e8427da68078b863ac38bf P e5f6c1ab1def4977b2ef5a61ae7c7b0a [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:38:21.856813 10934 raft_consensus.cc:3037] T 1236a038d8e8427da68078b863ac38bf P 1cb09abcfb6d42a4a841e6c77b8fb18a [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:38:21.857697 11068 raft_consensus.cc:2454] T 1236a038d8e8427da68078b863ac38bf P e5f6c1ab1def4977b2ef5a61ae7c7b0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a24847929dc7479095ffc1b0520bdc3c in term 1.
I20260619 14:38:21.857782 10934 raft_consensus.cc:2454] T 1236a038d8e8427da68078b863ac38bf P 1cb09abcfb6d42a4a841e6c77b8fb18a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a24847929dc7479095ffc1b0520bdc3c in term 1.
I20260619 14:38:21.857941 10733 leader_election.cc:302] T 1236a038d8e8427da68078b863ac38bf P a24847929dc7479095ffc1b0520bdc3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a24847929dc7479095ffc1b0520bdc3c, e5f6c1ab1def4977b2ef5a61ae7c7b0a; no voters:
I20260619 14:38:21.858090 11137 raft_consensus.cc:2781] T 1236a038d8e8427da68078b863ac38bf P a24847929dc7479095ffc1b0520bdc3c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:38:21.858280 11137 raft_consensus.cc:686] T 1236a038d8e8427da68078b863ac38bf P a24847929dc7479095ffc1b0520bdc3c [term 1 LEADER]: Becoming Leader. State: Replica: a24847929dc7479095ffc1b0520bdc3c, State: Running, Role: LEADER
I20260619 14:38:21.858577 11137 consensus_queue.cc:237] T 1236a038d8e8427da68078b863ac38bf P a24847929dc7479095ffc1b0520bdc3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a24847929dc7479095ffc1b0520bdc3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } } peers { permanent_uuid: "e5f6c1ab1def4977b2ef5a61ae7c7b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46113 } } peers { permanent_uuid: "1cb09abcfb6d42a4a841e6c77b8fb18a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42967 } }
I20260619 14:38:21.861619 10654 catalog_manager.cc:5495] T 1236a038d8e8427da68078b863ac38bf P a24847929dc7479095ffc1b0520bdc3c reported cstate change: term changed from 0 to 1, leader changed from <none> to a24847929dc7479095ffc1b0520bdc3c (127.0.98.1). New cstate: current_term: 1 leader_uuid: "a24847929dc7479095ffc1b0520bdc3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a24847929dc7479095ffc1b0520bdc3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5f6c1ab1def4977b2ef5a61ae7c7b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cb09abcfb6d42a4a841e6c77b8fb18a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42967 } health_report { overall_health: UNKNOWN } } }
I20260619 14:38:21.880499 11126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:21.877131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55374 (local address 127.0.98.1:42881)
0619 14:38:21.877255 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 14:38:21.877259 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:38:21.877296 (+ 37us) server_negotiation.cc:407] Connection header received
0619 14:38:21.877366 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:38:21.877368 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:38:21.877420 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:38:21.877507 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:38:21.877753 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:21.878239 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:21.879103 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:38:21.879320 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:38:21.879407 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:38:21.879702 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:38:21.880118 (+ 416us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:38:21.880228 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 14:38:21.880310 (+ 82us) server_negotiation.cc:299] Negotiation successful
0619 14:38:21.880365 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 14:38:21.886173 10934 raft_consensus.cc:1261] T 1236a038d8e8427da68078b863ac38bf P 1cb09abcfb6d42a4a841e6c77b8fb18a [term 1 FOLLOWER]: Refusing update from remote peer a24847929dc7479095ffc1b0520bdc3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:38:21.886173 11068 raft_consensus.cc:1261] T 1236a038d8e8427da68078b863ac38bf P e5f6c1ab1def4977b2ef5a61ae7c7b0a [term 1 FOLLOWER]: Refusing update from remote peer a24847929dc7479095ffc1b0520bdc3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:38:21.886881 11137 consensus_queue.cc:1035] T 1236a038d8e8427da68078b863ac38bf P a24847929dc7479095ffc1b0520bdc3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5f6c1ab1def4977b2ef5a61ae7c7b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:38:21.887079 11142 consensus_queue.cc:1035] T 1236a038d8e8427da68078b863ac38bf P a24847929dc7479095ffc1b0520bdc3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb09abcfb6d42a4a841e6c77b8fb18a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:38:21.894595 11147 mvcc.cc:205] Tried to move back new op lower bound from 7298580078118780928 to 7298580078015832064. Current Snapshot: MvccSnapshot[applied={T|T < 7298580078118780928}]
I20260619 14:38:21.901880 10653 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:38:21.902050 10653 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:38:21.904454 10653 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aa85dd2f985d4f8290c3bc9d7e0e4785: Sending DeleteTablet for 3 replicas of tablet 1236a038d8e8427da68078b863ac38bf
I20260619 14:38:21.905380 10780 tablet_service.cc:1514] Processing DeleteTablet for tablet 1236a038d8e8427da68078b863ac38bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:38:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55360
I20260619 14:38:21.905375 10914 tablet_service.cc:1514] Processing DeleteTablet for tablet 1236a038d8e8427da68078b863ac38bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:38:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43156
I20260619 14:38:21.905946 11155 tablet_replica.cc:330] T 1236a038d8e8427da68078b863ac38bf P 1cb09abcfb6d42a4a841e6c77b8fb18a: stopping tablet replica
I20260619 14:38:21.906091 11048 tablet_service.cc:1514] Processing DeleteTablet for tablet 1236a038d8e8427da68078b863ac38bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:38:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58516
I20260619 14:38:21.906436 11155 raft_consensus.cc:2229] T 1236a038d8e8427da68078b863ac38bf P 1cb09abcfb6d42a4a841e6c77b8fb18a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:38:21.906481 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 10717
I20260619 14:38:21.907363 11155 raft_consensus.cc:2258] T 1236a038d8e8427da68078b863ac38bf P 1cb09abcfb6d42a4a841e6c77b8fb18a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:38:21.907953 11157 tablet_replica.cc:330] T 1236a038d8e8427da68078b863ac38bf P e5f6c1ab1def4977b2ef5a61ae7c7b0a: stopping tablet replica
I20260619 14:38:21.908249 11157 raft_consensus.cc:2229] T 1236a038d8e8427da68078b863ac38bf P e5f6c1ab1def4977b2ef5a61ae7c7b0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:38:21.908540 11157 raft_consensus.cc:2258] T 1236a038d8e8427da68078b863ac38bf P e5f6c1ab1def4977b2ef5a61ae7c7b0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:38:21.909442 11155 ts_tablet_manager.cc:1905] T 1236a038d8e8427da68078b863ac38bf P 1cb09abcfb6d42a4a841e6c77b8fb18a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:38:21.910025 11157 ts_tablet_manager.cc:1905] T 1236a038d8e8427da68078b863ac38bf P e5f6c1ab1def4977b2ef5a61ae7c7b0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:38:21.911528 11155 ts_tablet_manager.cc:1918] T 1236a038d8e8427da68078b863ac38bf P 1cb09abcfb6d42a4a841e6c77b8fb18a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:38:21.911607 11155 log.cc:1196] T 1236a038d8e8427da68078b863ac38bf P 1cb09abcfb6d42a4a841e6c77b8fb18a: Deleting WAL directory at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/ts-1/wal/wals/1236a038d8e8427da68078b863ac38bf
I20260619 14:38:21.911854 11155 ts_tablet_manager.cc:1939] T 1236a038d8e8427da68078b863ac38bf P 1cb09abcfb6d42a4a841e6c77b8fb18a: Deleting consensus metadata
I20260619 14:38:21.912927 11157 ts_tablet_manager.cc:1918] T 1236a038d8e8427da68078b863ac38bf P e5f6c1ab1def4977b2ef5a61ae7c7b0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:38:21.912954 10641 catalog_manager.cc:4841] TS 1cb09abcfb6d42a4a841e6c77b8fb18a (127.0.98.2:42967): tablet 1236a038d8e8427da68078b863ac38bf (table test-table [id=5c7713cfb6154297b8e074e61ec94871]) successfully deleted
I20260619 14:38:21.913110 11157 log.cc:1196] T 1236a038d8e8427da68078b863ac38bf P e5f6c1ab1def4977b2ef5a61ae7c7b0a: Deleting WAL directory at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781879837648036-392-0/minicluster-data/ts-2/wal/wals/1236a038d8e8427da68078b863ac38bf
I20260619 14:38:21.913419 11157 ts_tablet_manager.cc:1939] T 1236a038d8e8427da68078b863ac38bf P e5f6c1ab1def4977b2ef5a61ae7c7b0a: Deleting consensus metadata
W20260619 14:38:21.913844 10641 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42881 (error 108)
I20260619 14:38:21.914563 10639 catalog_manager.cc:4841] TS e5f6c1ab1def4977b2ef5a61ae7c7b0a (127.0.98.3:46113): tablet 1236a038d8e8427da68078b863ac38bf (table test-table [id=5c7713cfb6154297b8e074e61ec94871]) successfully deleted
I20260619 14:38:21.914682 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 10851
I20260619 14:38:21.914841 11125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:38:21.914580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42881 (local address 127.0.0.1:55378)
0619 14:38:21.914694 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0619 14:38:21.914763 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260619 14:38:21.915818 10641 catalog_manager.cc:4584] TS a24847929dc7479095ffc1b0520bdc3c (127.0.98.1:42881): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1236a038d8e8427da68078b863ac38bf: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42881: connect: Connection refused (error 111)
I20260619 14:38:21.919580 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 10985
I20260619 14:38:21.923967 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 10623
2026-06-19T14:38:21Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2170 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:38:21 dist-test-slave-32hj krb5kdc[11161](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:38:21 dist-test-slave-32hj krb5kdc[11161](info): set up 2 sockets
Jun 19 14:38:21 dist-test-slave-32hj krb5kdc[11161](info): commencing operation
krb5kdc: starting...
W20260619 14:38:22.885476 392 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/krb5kdc/test-user.keytab.
Jun 19 14:38:22 dist-test-slave-32hj krb5kdc[11161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:38:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:38:22Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260619 14:38:23.051271 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task7PHJY_/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/postgres -l logfile start
2026-06-19 14:38:25.561 UTC [11191] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 14:38:25.561 UTC [11191] LOG: listening on IPv4 address "127.0.98.20", port 47237
2026-06-19 14:38:25.566 UTC [11191] LOG: listening on Unix socket "/tmp/.s.PGSQL.47237"
2026-06-19 14:38:25.594 UTC [11194] LOG: database system was shut down at 2026-06-19 14:38:24 UTC
2026-06-19 14:38:25.598 UTC [11191] LOG: database system is ready to accept connections
I20260619 14:38:26.461333 392 mini_postgres.cc:96] Postgres bound to 47237
2026-06-19 14:38:26.467 UTC [11202] FATAL: database "slave" does not exist
127.0.98.20:47237 - accepting connections
I20260619 14:38:26.467854 392 mini_ranger.cc:161] Starting Ranger...
I20260619 14:38:26.482415 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 14:38:27.000909 392 mini_ranger.cc:87] Created ranger Postgres database
I20260619 14:38:27.001008 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ranger-admin
2026-06-19 14:38:27,268 [I] DB FLAVOR :POSTGRES
2026-06-19 14:38:27,269 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 14:38:27,269 [I] Checking connection..
2026-06-19 14:38:27,269 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7PHJY_/build/debug/bin/postgresql.jar:/tmp/dist-test-task7PHJY_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 14:38:27,525 [I] Checking connection passed.
2026-06-19 14:38:27,525 [I] --------- Verifying version history table ---------
2026-06-19 14:38:27,525 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7PHJY_/build/debug/bin/postgresql.jar:/tmp/dist-test-task7PHJY_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 14:38:27,764 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 14:38:27,764 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7PHJY_/build/debug/bin/postgresql.jar:/tmp/dist-test-task7PHJY_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 14:38:27,994 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 14:38:27,995 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 14:38:27,995 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7PHJY_/build/debug/bin/postgresql.jar:/tmp/dist-test-task7PHJY_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47237/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7PHJY_/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 14:38:28.250 UTC [11266] WARNING: there is no transaction in progress
2026-06-19 14:38:28,255 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 14:38:28,255 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7PHJY_/build/debug/bin/postgresql.jar:/tmp/dist-test-task7PHJY_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 14:38:28,507 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 14:38:28,507 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 14:38:28,507 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7PHJY_/build/debug/bin/postgresql.jar:/tmp/dist-test-task7PHJY_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 14:38:28,756 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7PHJY_/build/debug/bin/postgresql.jar:/tmp/dist-test-task7PHJY_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 14:38:29,005 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7PHJY_/build/debug/bin/postgresql.jar:/tmp/dist-test-task7PHJY_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-32hj.c.gcp-upstream.internal','N') ;"
2026-06-19 14:38:29,230 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7PHJY_/build/debug/bin/postgresql.jar:/tmp/dist-test-task7PHJY_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 14:38:29,452 [I] Table x_portal_user does not exist in database ranger
2026-06-19 14:38:29,453 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7PHJY_/build/debug/bin/postgresql.jar:/tmp/dist-test-task7PHJY_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 14:38:29,690 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 14:38:29,691 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7PHJY_/build/debug/bin/postgresql.jar:/tmp/dist-test-task7PHJY_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 14:38:29,926 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7PHJY_/build/debug/bin/postgresql.jar:/tmp/dist-test-task7PHJY_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 14:38:30,164 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 14:38:30,165 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7PHJY_/build/debug/bin/postgresql.jar:/tmp/dist-test-task7PHJY_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47237/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7PHJY_/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 14:38:31.129 UTC [11374] WARNING: there is no transaction in progress
2026-06-19 14:38:31.145 UTC [11374] WARNING: there is no transaction in progress
2026-06-19 14:38:31.161 UTC [11374] WARNING: there is no transaction in progress
2026-06-19 14:38:31.177 UTC [11374] WARNING: there is no transaction in progress
2026-06-19 14:38:31.193 UTC [11374] WARNING: there is no transaction in progress
2026-06-19 14:38:31.211 UTC [11374] WARNING: there is no transaction in progress
2026-06-19 14:38:31.267 UTC [11374] WARNING: there is no transaction in progress
2026-06-19 14:38:31.273 UTC [11374] WARNING: there is no transaction in progress
2026-06-19 14:38:31.287 UTC [11374] WARNING: there is no transaction in progress
2026-06-19 14:38:31.297 UTC [11374] WARNING: there is no transaction in progress
2026-06-19 14:38:31.309 UTC [11374] WARNING: there is no transaction in progress
2026-06-19 14:38:31.325 UTC [11374] WARNING: there is no transaction in progress
2026-06-19 14:38:31.335 UTC [11374] WARNING: there is no transaction in progress
2026-06-19 14:38:31.342 UTC [11374] WARNING: there is no transaction in progress
2026-06-19 14:38:31.348 UTC [11374] WARNING: there is no transaction in progress
2026-06-19 14:38:31.354 UTC [11374] WARNING: there is no transaction in progress
2026-06-19 14:38:31,955 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 14:38:31,955 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7PHJY_/build/debug/bin/postgresql.jar:/tmp/dist-test-task7PHJY_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 14:38:32,178 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 14:38:32,179 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7PHJY_/build/debug/bin/postgresql.jar:/tmp/dist-test-task7PHJY_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 14:38:32,430 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 14:38:32,430 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7PHJY_/build/debug/bin/postgresql.jar:/tmp/dist-test-task7PHJY_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 14:38:32,681 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 14:38:32,681 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7PHJY_/build/debug/bin/postgresql.jar:/tmp/dist-test-task7PHJY_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 14:38:32,916 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7PHJY_/build/debug/bin/postgresql.jar:/tmp/dist-test-task7PHJY_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 14:38:33,151 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7PHJY_/build/debug/bin/postgresql.jar:/tmp/dist-test-task7PHJY_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-32hj.c.gcp-upstream.internal';"
2026-06-19 14:38:33,395 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 14:38:33,395 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7PHJY_/build/debug/bin/postgresql.jar:/tmp/dist-test-task7PHJY_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 14:38:33,646 [I] DB_PATCHES have already been applied
I20260619 14:38:33.651373 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ranger-admin:/tmp/dist-test-task7PHJY_/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task7PHJY_/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task7PHJY_/build/debug/bin/postgresql.jar:/tmp/dist-test-task7PHJY_/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 14:38:33.651433 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260619 14:38:33.654546 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:38703
Jun 19, 2026 2:38:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 2:38:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 2:38:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 2:38:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 2:38:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 2:38:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 2:38:34 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 14:38:34.648073 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.180s user 0.000s sys 0.009s
Jun 19, 2026 2:38:54 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 2:38:54 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Jun 19, 2026 2:38:54 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 2:38:54 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 2:38:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 2:38:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 2:38:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 2:38:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 2:38:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 2:38:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 2:38:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 2:38:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 2:38:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 2:38:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 2:38:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 2:38:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 2:38:54 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 2:38:55 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 14:38:55.941460 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 14:38:56 dist-test-slave-32hj krb5kdc[11161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879936, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 14:38:56.018910 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 14:38:56.027724 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 14:38:56.724422 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 14:38:56.724502 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ranger-kms
I20260619 14:38:56.724586 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:47237
2026-06-19 14:38:56,884 [I] DB FLAVOR :POSTGRES
2026-06-19 14:38:56,884 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 14:38:56,884 [I] Checking connection
2026-06-19 14:38:56,884 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7PHJY_/build/debug/bin/postgresql.jar:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47237/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 14:38:57,129 [I] connection success
2026-06-19 14:38:57,129 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 14:38:57,129 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 14:38:57,129 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7PHJY_/build/debug/bin/postgresql.jar:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47237/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 14:38:57,355 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 14:38:57,356 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 14:38:57,356 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 14:38:57,356 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7PHJY_/build/debug/bin/postgresql.jar:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:47237/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 14:38:57,588 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 14:38:58.016204 392 mini_ranger_kms.cc:310] Created kms service
I20260619 14:38:58.174088 392 mini_ranger_kms.cc:326] Created kudu user
I20260619 14:38:58.213135 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 14:38:58 dist-test-slave-32hj krb5kdc[11161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879936, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 14:38:58.573707 392 mini_ranger_kms.cc:384] Added ranger policy
I20260619 14:38:58.573997 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ranger-kms:/tmp/dist-test-task7PHJY_/build/debug/bin/postgresql.jar:/tmp/dist-test-task7PHJY_/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task7PHJY_/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 14:38:58.574020 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260619 14:38:58.577466 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11554
I20260619 14:38:58.577596 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:55533
Jun 19, 2026 2:38:59 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 2:38:59 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 2:38:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 2:38:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 2:38:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 2:38:59 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 14:39:00.536484 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.518s user 0.011s sys 0.132s
I20260619 14:39:00.536610 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 14:39:00.536737 392 mini_ranger_kms.cc:401] 127.0.98.20:55533/kms/v1/keys
Jun 19 14:39:00 dist-test-slave-32hj krb5kdc[11161](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781879940, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 14:39:01 dist-test-slave-32hj krb5kdc[11161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879940, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 19 14:39:01 dist-test-slave-32hj krb5kdc[11161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879940, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 19 14:39:03 dist-test-slave-32hj krb5kdc[11161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:39:03.683661 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36665
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36665
--ranger_config_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55533/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:39:03.793376 11605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:39:03.793644 11605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:39:03.793680 11605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:39:03.797704 11605 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:39:03.797770 11605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:39:03.797787 11605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:39:03.797803 11605 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:39:03.797816 11605 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:39:03.803207 11605 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:55533/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36665
--ranger_config_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11605
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:39:03.804220 11605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:39:03.805258 11605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:39:03.813334 11611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:39:03.813349 11610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:39:03.813427 11613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:39:03.813716 11605 server_base.cc:1034] running on GCE node
I20260619 14:39:03.814240 11605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:39:03.815280 11605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:39:03.816478 11605 hybrid_clock.cc:648] HybridClock initialized: now 1781879943816456 us; error 35 us; skew 500 ppm
Jun 19 14:39:03 dist-test-slave-32hj krb5kdc[11161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879943, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:39:03.819342 11605 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:39:03.820428 11605 webserver.cc:460] Webserver started at http://127.0.98.62:45157/ using document root <none> and password file <none>
I20260619 14:39:03.821102 11605 fs_manager.cc:359] Metadata directory not provided
I20260619 14:39:03.821182 11605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:39:03.821409 11605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 14:39:03 dist-test-slave-32hj krb5kdc[11161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879943, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 14:39:03.867764 11605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/master-0/data/instance:
uuid: "9fa9c20067164da7b786530e76b84604"
format_stamp: "Formatted at 2026-06-19 14:39:03 on dist-test-slave-32hj"
server_key: "0204854028bb4081db22e5b8335ca299"
server_key_iv: "0c362a57e2ceb9f2b2116742097170af"
server_key_version: "kuduclusterkey@0"
I20260619 14:39:03.868531 11605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/master-0/wal/instance:
uuid: "9fa9c20067164da7b786530e76b84604"
format_stamp: "Formatted at 2026-06-19 14:39:03 on dist-test-slave-32hj"
server_key: "0204854028bb4081db22e5b8335ca299"
server_key_iv: "0c362a57e2ceb9f2b2116742097170af"
server_key_version: "kuduclusterkey@0"
I20260619 14:39:03.873356 11605 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 14:39:03.889283 11620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:39:03.890626 11605 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:39:03.890741 11605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/master-0/wal
uuid: "9fa9c20067164da7b786530e76b84604"
format_stamp: "Formatted at 2026-06-19 14:39:03 on dist-test-slave-32hj"
server_key: "0204854028bb4081db22e5b8335ca299"
server_key_iv: "0c362a57e2ceb9f2b2116742097170af"
server_key_version: "kuduclusterkey@0"
I20260619 14:39:03.890863 11605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:39:03.903909 11605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:39:03.906489 11605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:39:03.906657 11605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:39:03.913760 11605 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36665
I20260619 14:39:03.913790 11672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36665 every 8 connection(s)
I20260619 14:39:03.914675 11605 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:39:03.917409 11673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:39:03.920174 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 11605
I20260619 14:39:03.920303 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/master-0/wal/instance
I20260619 14:39:03.920610 392 external_mini_cluster.cc:1428] Setting key 282eaf6a02916aabf108cf92197688b3
I20260619 14:39:03.923544 11673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fa9c20067164da7b786530e76b84604: Bootstrap starting.
I20260619 14:39:03.925748 11673 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9fa9c20067164da7b786530e76b84604: Neither blocks nor log segments found. Creating new log.
I20260619 14:39:03.926436 11673 log.cc:824] T 00000000000000000000000000000000 P 9fa9c20067164da7b786530e76b84604: Log is configured to *not* fsync() on all Append() calls
I20260619 14:39:03.928314 11673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fa9c20067164da7b786530e76b84604: No bootstrap required, opened a new log
Jun 19 14:39:03 dist-test-slave-32hj krb5kdc[11161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:39:03.931962 11673 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9fa9c20067164da7b786530e76b84604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa9c20067164da7b786530e76b84604" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36665 } }
I20260619 14:39:03.932185 11673 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9fa9c20067164da7b786530e76b84604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:39:03.932221 11673 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9fa9c20067164da7b786530e76b84604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fa9c20067164da7b786530e76b84604, State: Initialized, Role: FOLLOWER
I20260619 14:39:03.932642 11673 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fa9c20067164da7b786530e76b84604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa9c20067164da7b786530e76b84604" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36665 } }
I20260619 14:39:03.932792 11673 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9fa9c20067164da7b786530e76b84604 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:39:03.932857 11673 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9fa9c20067164da7b786530e76b84604 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:39:03.932946 11673 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9fa9c20067164da7b786530e76b84604 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:39:03.933998 11673 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9fa9c20067164da7b786530e76b84604 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa9c20067164da7b786530e76b84604" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36665 } }
I20260619 14:39:03.934325 11676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:03.922064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58276 (local address 127.0.98.62:36665)
0619 14:39:03.922490 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0619 14:39:03.922511 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 14:39:03.922546 (+ 35us) server_negotiation.cc:407] Connection header received
0619 14:39:03.923158 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:39:03.923174 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:39:03.923500 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:39:03.923852 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:39:03.924620 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:03.925635 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:03.926508 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:03.927124 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:03.927274 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:03.929761 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:39:03.929767 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:39:03.929779 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:39:03.929805 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:39:03.931313 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:03.931711 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:03.931717 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:03.931723 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:03.931826 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:03.932084 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:03.932087 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:03.932089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:03.932559 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:39:03.932787 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0619 14:39:03.933088 (+ 301us) server_negotiation.cc:299] Negotiation successful
0619 14:39:03.933376 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":132,"threads_started":1}
I20260619 14:39:03.934338 11673 leader_election.cc:302] T 00000000000000000000000000000000 P 9fa9c20067164da7b786530e76b84604 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9fa9c20067164da7b786530e76b84604; no voters:
I20260619 14:39:03.934695 11673 leader_election.cc:288] T 00000000000000000000000000000000 P 9fa9c20067164da7b786530e76b84604 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:39:03.934783 11678 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9fa9c20067164da7b786530e76b84604 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:39:03.934999 11678 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9fa9c20067164da7b786530e76b84604 [term 1 LEADER]: Becoming Leader. State: Replica: 9fa9c20067164da7b786530e76b84604, State: Running, Role: LEADER
I20260619 14:39:03.935331 11678 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fa9c20067164da7b786530e76b84604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa9c20067164da7b786530e76b84604" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36665 } }
I20260619 14:39:03.935866 11673 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9fa9c20067164da7b786530e76b84604 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:39:03.937201 11679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fa9c20067164da7b786530e76b84604 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9fa9c20067164da7b786530e76b84604" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa9c20067164da7b786530e76b84604" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36665 } } }
I20260619 14:39:03.937314 11679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fa9c20067164da7b786530e76b84604 [sys.catalog]: This master's current role is: LEADER
I20260619 14:39:03.938024 11680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fa9c20067164da7b786530e76b84604 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fa9c20067164da7b786530e76b84604. Latest consensus state: current_term: 1 leader_uuid: "9fa9c20067164da7b786530e76b84604" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa9c20067164da7b786530e76b84604" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36665 } } }
I20260619 14:39:03.938118 11680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fa9c20067164da7b786530e76b84604 [sys.catalog]: This master's current role is: LEADER
I20260619 14:39:03.938843 11689 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:39:03.939072 11673 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 14:39:03.944208 11689 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:39:03.951050 11689 catalog_manager.cc:1347] Generated new cluster ID: f7a59dbb6b404f1d808799e1f2e639dc
I20260619 14:39:03.951126 11689 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:39:03.965078 11689 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:39:03.966008 11689 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:39:03.977188 11689 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9fa9c20067164da7b786530e76b84604: Generated new TSK 0
I20260619 14:39:03.977941 11689 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 14:39:04.679945 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55533/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36665
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:39:04.822140 11718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:39:04.822527 11718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:39:04.822592 11718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:39:04.827183 11718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:39:04.827242 11718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:39:04.827322 11718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:39:04.832590 11718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:55533/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36665
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:39:04.833722 11718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:39:04.834784 11718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:39:04.844321 11725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:39:04.844463 11726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:39:04.844353 11728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:39:04.844841 11718 server_base.cc:1034] running on GCE node
I20260619 14:39:04.845259 11718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:39:04.845979 11718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:39:04.847133 11718 hybrid_clock.cc:648] HybridClock initialized: now 1781879944847100 us; error 52 us; skew 500 ppm
Jun 19 14:39:04 dist-test-slave-32hj krb5kdc[11161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879944, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:39:04.849900 11718 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:39:04.851019 11718 webserver.cc:460] Webserver started at http://127.0.98.1:41639/ using document root <none> and password file <none>
I20260619 14:39:04.851712 11718 fs_manager.cc:359] Metadata directory not provided
I20260619 14:39:04.851804 11718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:39:04.852039 11718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 14:39:04 dist-test-slave-32hj krb5kdc[11161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879944, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 14:39:04.868351 11718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ts-0/data/instance:
uuid: "e52d2479e3fe45f79121b006c283d330"
format_stamp: "Formatted at 2026-06-19 14:39:04 on dist-test-slave-32hj"
server_key: "8f9e4f9d055985a12aa8a25d1f79513d"
server_key_iv: "68916419194946d12ed66dc9d3662590"
server_key_version: "kuduclusterkey@0"
I20260619 14:39:04.870163 11718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e52d2479e3fe45f79121b006c283d330"
format_stamp: "Formatted at 2026-06-19 14:39:04 on dist-test-slave-32hj"
server_key: "8f9e4f9d055985a12aa8a25d1f79513d"
server_key_iv: "68916419194946d12ed66dc9d3662590"
server_key_version: "kuduclusterkey@0"
I20260619 14:39:04.886546 11718 fs_manager.cc:691] Time spent creating directory manager: real 0.016s user 0.004s sys 0.000s
I20260619 14:39:04.913165 11737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:39:04.914850 11718 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260619 14:39:04.914984 11718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ts-0/wal
uuid: "e52d2479e3fe45f79121b006c283d330"
format_stamp: "Formatted at 2026-06-19 14:39:04 on dist-test-slave-32hj"
server_key: "8f9e4f9d055985a12aa8a25d1f79513d"
server_key_iv: "68916419194946d12ed66dc9d3662590"
server_key_version: "kuduclusterkey@0"
I20260619 14:39:04.915112 11718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:39:04.938189 11718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:39:04.941188 11718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:39:04.941367 11718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:39:04.941936 11718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:39:04.942816 11718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:39:04.942864 11718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:39:04.942958 11718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:39:04.942991 11718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:39:04.952488 11718 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35781
I20260619 14:39:04.953670 11718 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:39:04.954075 11850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35781 every 8 connection(s)
I20260619 14:39:04.960343 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 11718
I20260619 14:39:04.960456 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:39:04.960815 392 external_mini_cluster.cc:1428] Setting key a5b465b72f73af8b0082887735537b17
Jun 19 14:39:04 dist-test-slave-32hj krb5kdc[11161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879944, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 14:39:04.973521 11853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:04.956303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42693 (local address 127.0.98.62:36665)
0619 14:39:04.960660 (+ 4357us) server_negotiation.cc:206] Beginning negotiation
0619 14:39:04.960668 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:39:04.960685 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:39:04.960751 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:39:04.960754 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:39:04.960806 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:39:04.960906 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:39:04.962723 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:04.963686 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:04.965347 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:04.965662 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:04.965752 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:04.969199 (+ 3447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:39:04.969204 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:39:04.969206 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:39:04.969233 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:39:04.971038 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:04.971677 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:04.971680 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:04.971681 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:04.971756 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:04.972165 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:04.972168 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:04.972170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:04.972401 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:39:04.972499 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 14:39:04.973194 (+ 695us) server_negotiation.cc:299] Negotiation successful
0619 14:39:04.973324 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4265,"thread_start_us":107,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 14:39:04.974638 11854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:04.956792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36665 (local address 127.0.98.1:42693)
0619 14:39:04.957249 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:04.957300 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 14:39:04.958113 (+ 813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:39:04.961439 (+ 3326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:39:04.961450 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:39:04.961799 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:39:04.962557 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:04.962578 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:04.963824 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:04.963828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:04.965204 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:04.965215 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:04.965921 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:04.965924 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:04.966113 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:04.966792 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:39:04.966825 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:39:04.968992 (+ 2167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:39:04.971176 (+ 2184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:39:04.971182 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:39:04.971200 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:39:04.971559 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:39:04.971846 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:39:04.971850 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:39:04.971852 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:39:04.972049 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:39:04.972569 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:39:04.972575 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:39:04.972944 (+ 369us) client_negotiation.cc:769] Sending connection context
0619 14:39:04.973118 (+ 174us) client_negotiation.cc:240] Negotiation successful
0619 14:39:04.973442 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":106,"threads_started":1}
I20260619 14:39:04.976233 11851 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36665
I20260619 14:39:04.976471 11851 heartbeater.cc:464] Registering TS with master...
I20260619 14:39:04.977054 11851 heartbeater.cc:511] Master 127.0.98.62:36665 requested a full tablet report, sending...
I20260619 14:39:04.978605 11637 ts_manager.cc:194] Registered new tserver with Master: e52d2479e3fe45f79121b006c283d330 (127.0.98.1:35781)
I20260619 14:39:04.979712 11637 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42693
Jun 19 14:39:04 dist-test-slave-32hj krb5kdc[11161](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781879944, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 14:39:05.002499 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55533/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36665
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:39:05.161334 11857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:39:05.161744 11857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:39:05.161873 11857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:39:05.167922 11857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:39:05.168001 11857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:39:05.168089 11857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:39:05.175951 11857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:55533/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36665
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:39:05.177359 11857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:39:05.178685 11857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:39:05.190822 11863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:39:05.190837 11864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:39:05.191424 11866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:39:05.191743 11857 server_base.cc:1034] running on GCE node
I20260619 14:39:05.192243 11857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:39:05.193095 11857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:39:05.194298 11857 hybrid_clock.cc:648] HybridClock initialized: now 1781879945194271 us; error 38 us; skew 500 ppm
Jun 19 14:39:05 dist-test-slave-32hj krb5kdc[11161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879945, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:39:05.197950 11857 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:39:05.199033 11857 webserver.cc:460] Webserver started at http://127.0.98.2:45759/ using document root <none> and password file <none>
I20260619 14:39:05.199774 11857 fs_manager.cc:359] Metadata directory not provided
I20260619 14:39:05.199906 11857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:39:05.200186 11857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 14:39:05 dist-test-slave-32hj krb5kdc[11161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879945, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 14:39:05.216675 11857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ts-1/data/instance:
uuid: "b9b391eb67904b468f986e432f881149"
format_stamp: "Formatted at 2026-06-19 14:39:05 on dist-test-slave-32hj"
server_key: "388c10fee2ada966d048b91b0704d365"
server_key_iv: "134dd8cdc87fa41de8c1fc991ac7ff3f"
server_key_version: "kuduclusterkey@0"
I20260619 14:39:05.217600 11857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b9b391eb67904b468f986e432f881149"
format_stamp: "Formatted at 2026-06-19 14:39:05 on dist-test-slave-32hj"
server_key: "388c10fee2ada966d048b91b0704d365"
server_key_iv: "134dd8cdc87fa41de8c1fc991ac7ff3f"
server_key_version: "kuduclusterkey@0"
I20260619 14:39:05.226181 11857 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.000s
I20260619 14:39:05.241269 11873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:39:05.242959 11857 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 14:39:05.243086 11857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ts-1/wal
uuid: "b9b391eb67904b468f986e432f881149"
format_stamp: "Formatted at 2026-06-19 14:39:05 on dist-test-slave-32hj"
server_key: "388c10fee2ada966d048b91b0704d365"
server_key_iv: "134dd8cdc87fa41de8c1fc991ac7ff3f"
server_key_version: "kuduclusterkey@0"
I20260619 14:39:05.243225 11857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:39:05.269543 11857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:39:05.273378 11857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:39:05.273581 11857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:39:05.274382 11857 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:39:05.275496 11857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:39:05.275560 11857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:39:05.275615 11857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:39:05.275652 11857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260619 14:39:05.286391 11857 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45227
I20260619 14:39:05.287619 11857 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:39:05.289005 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 11857
I20260619 14:39:05.289119 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:39:05.289395 392 external_mini_cluster.cc:1428] Setting key 12a63ad4c887834cfa6293312d2ef94f
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 14:39:05.320580 11986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45227 every 8 connection(s)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Jun 19 14:39:05 dist-test-slave-32hj krb5kdc[11161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879945, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:39:05.331975 11853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:05.292048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35681 (local address 127.0.98.62:36665)
0619 14:39:05.292910 (+ 862us) server_negotiation.cc:206] Beginning negotiation
0619 14:39:05.292915 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:39:05.303053 (+ 10138us) server_negotiation.cc:407] Connection header received
0619 14:39:05.304465 (+ 1412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:39:05.304469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:39:05.304530 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:39:05.304590 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:39:05.309375 (+ 4785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:05.310243 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:05.315974 (+ 5731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:05.316344 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:05.316454 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:05.324594 (+ 8140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:39:05.324599 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:39:05.324602 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:39:05.324639 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:39:05.326437 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:05.329913 (+ 3476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:05.329918 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:05.329920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:05.330033 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:05.330492 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:05.330496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:05.330498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:05.330731 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:39:05.330808 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 14:39:05.331618 (+ 810us) server_negotiation.cc:299] Negotiation successful
0619 14:39:05.331770 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":761}
I20260619 14:39:05.333945 11989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:05.291015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36665 (local address 127.0.98.2:35681)
0619 14:39:05.302827 (+ 11812us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:05.302914 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0619 14:39:05.304148 (+ 1234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:39:05.305107 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:39:05.305118 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:39:05.305543 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:39:05.309160 (+ 3617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:05.309184 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:05.310432 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:05.310435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:05.311964 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:05.311976 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:05.320953 (+ 8977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:05.320958 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:05.321191 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:05.322288 (+ 1097us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:39:05.322321 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:39:05.324323 (+ 2002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:39:05.327074 (+ 2751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:39:05.327081 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:39:05.327102 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:39:05.327476 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:39:05.330146 (+ 2670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:39:05.330150 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:39:05.330152 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:39:05.330368 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:39:05.330875 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:39:05.330883 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:39:05.331294 (+ 411us) client_negotiation.cc:769] Sending connection context
0619 14:39:05.332477 (+ 1183us) client_negotiation.cc:240] Negotiation successful
0619 14:39:05.332878 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11511,"thread_start_us":124,"threads_started":1}
I20260619 14:39:05.335543 11987 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36665
I20260619 14:39:05.335786 11987 heartbeater.cc:464] Registering TS with master...
I20260619 14:39:05.336364 11987 heartbeater.cc:511] Master 127.0.98.62:36665 requested a full tablet report, sending...
I20260619 14:39:05.337612 11637 ts_manager.cc:194] Registered new tserver with Master: b9b391eb67904b468f986e432f881149 (127.0.98.2:45227)
I20260619 14:39:05.338378 11637 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35681
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 14:39:05.353016 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55533/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36665
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:39:05 dist-test-slave-32hj krb5kdc[11161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879944, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260619 14:39:05.518087 11992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:39:05.518529 11992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:39:05.518606 11992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:39:05.524946 11992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:39:05.525017 11992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:39:05.525097 11992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:39:05.532548 11992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:55533/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36665
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:39:05.534109 11992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:39:05.536755 11992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:39:05.549307 12000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:39:05.550732 11997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:39:05.552867 11998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:39:05.557893 11992 server_base.cc:1034] running on GCE node
I20260619 14:39:05.558533 11992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:39:05.559347 11992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:39:05.574812 11992 hybrid_clock.cc:648] HybridClock initialized: now 1781879945574724 us; error 99 us; skew 500 ppm
Jun 19 14:39:05 dist-test-slave-32hj krb5kdc[11161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879945, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:39:05.578640 11992 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:39:05.579843 11992 webserver.cc:460] Webserver started at http://127.0.98.3:35123/ using document root <none> and password file <none>
I20260619 14:39:05.580574 11992 fs_manager.cc:359] Metadata directory not provided
I20260619 14:39:05.580945 11992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:39:05.581238 11992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 14:39:05 dist-test-slave-32hj krb5kdc[11161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879945, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 14:39:05.596576 11992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ts-2/data/instance:
uuid: "73d82159532149399ffd45a28df5ef62"
format_stamp: "Formatted at 2026-06-19 14:39:05 on dist-test-slave-32hj"
server_key: "15c14f140d754285787f61ce4b79274f"
server_key_iv: "9cf785efd4c433b2eeda26f8d1faf02f"
server_key_version: "kuduclusterkey@0"
I20260619 14:39:05.597379 11992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ts-2/wal/instance:
uuid: "73d82159532149399ffd45a28df5ef62"
format_stamp: "Formatted at 2026-06-19 14:39:05 on dist-test-slave-32hj"
server_key: "15c14f140d754285787f61ce4b79274f"
server_key_iv: "9cf785efd4c433b2eeda26f8d1faf02f"
server_key_version: "kuduclusterkey@0"
I20260619 14:39:05.610502 11992 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.004s sys 0.000s
I20260619 14:39:05.630326 12007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:39:05.632222 11992 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 14:39:05.632372 11992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ts-2/wal
uuid: "73d82159532149399ffd45a28df5ef62"
format_stamp: "Formatted at 2026-06-19 14:39:05 on dist-test-slave-32hj"
server_key: "15c14f140d754285787f61ce4b79274f"
server_key_iv: "9cf785efd4c433b2eeda26f8d1faf02f"
server_key_version: "kuduclusterkey@0"
I20260619 14:39:05.632524 11992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:39:05.670753 11992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:39:05.674623 11992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:39:05.674885 11992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:39:05.675679 11992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:39:05.676798 11992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:39:05.676913 11992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:39:05.677016 11992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:39:05.677078 11992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:39:05.690948 11992 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42381
I20260619 14:39:05.692440 11992 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:39:05.694140 12120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42381 every 8 connection(s)
I20260619 14:39:05.702270 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 11992
I20260619 14:39:05.702391 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:39:05.702661 392 external_mini_cluster.cc:1428] Setting key 3feb653e275f68af52554be461530d65
Jun 19 14:39:05 dist-test-slave-32hj krb5kdc[11161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879945, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:39:05.712179 11853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:05.696520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37253 (local address 127.0.98.62:36665)
0619 14:39:05.696643 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 14:39:05.696648 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:39:05.698303 (+ 1655us) server_negotiation.cc:407] Connection header received
0619 14:39:05.699473 (+ 1170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:39:05.699477 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:39:05.699530 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:39:05.699579 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:39:05.701122 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:05.701961 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:05.704935 (+ 2974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:05.705276 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:05.705343 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:05.708217 (+ 2874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:39:05.708221 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:39:05.708224 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:39:05.708250 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:39:05.709927 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:05.710480 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:05.710484 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:05.710485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:05.710563 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:05.710934 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:05.710938 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:05.710939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:05.711169 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:39:05.711253 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 14:39:05.711905 (+ 652us) server_negotiation.cc:299] Negotiation successful
0619 14:39:05.712019 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 14:39:05.713939 12123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:05.697036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36665 (local address 127.0.98.3:37253)
0619 14:39:05.698145 (+ 1109us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:05.698201 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 14:39:05.699239 (+ 1038us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:39:05.699963 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:39:05.699972 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:39:05.700315 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:39:05.700967 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:05.700984 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:05.703410 (+ 2426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:05.703413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:05.704801 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:05.704812 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:05.705529 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:05.705532 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:05.705726 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:05.706359 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:39:05.706378 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:39:05.708011 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:39:05.710047 (+ 2036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:39:05.710053 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:39:05.710068 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:39:05.710375 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:39:05.710655 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:39:05.710657 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:39:05.710659 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:39:05.710816 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:39:05.711320 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:39:05.711327 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:39:05.711637 (+ 310us) client_negotiation.cc:769] Sending connection context
0619 14:39:05.712584 (+ 947us) client_negotiation.cc:240] Negotiation successful
0619 14:39:05.712931 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":899,"thread_start_us":124,"threads_started":1}
I20260619 14:39:05.715359 12121 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36665
I20260619 14:39:05.715584 12121 heartbeater.cc:464] Registering TS with master...
I20260619 14:39:05.716110 12121 heartbeater.cc:511] Master 127.0.98.62:36665 requested a full tablet report, sending...
I20260619 14:39:05.717130 11637 ts_manager.cc:194] Registered new tserver with Master: 73d82159532149399ffd45a28df5ef62 (127.0.98.3:42381)
I20260619 14:39:05.717832 11637 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37253
I20260619 14:39:05.728430 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:39:05.740165 11853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:05.730790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58278 (local address 127.0.98.62:36665)
0619 14:39:05.730917 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 14:39:05.730921 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:39:05.731298 (+ 377us) server_negotiation.cc:407] Connection header received
0619 14:39:05.731510 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:39:05.731513 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:39:05.731559 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:39:05.731605 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:39:05.732617 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:05.733435 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:05.734690 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:05.735039 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:05.735113 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:05.737111 (+ 1998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:39:05.737115 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:39:05.737118 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:39:05.737162 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:39:05.738578 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:05.739097 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:05.739101 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:05.739103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:05.739172 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:05.739411 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:05.739415 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:05.739417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:05.739579 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:39:05.739691 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 14:39:05.739912 (+ 221us) server_negotiation.cc:299] Negotiation successful
0619 14:39:05.740014 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 14:39:05.742825 11637 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58278:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:39:05.746137 11637 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:39:05.765796 12134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:05.757274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42381 (local address 127.0.0.1:55172)
0619 14:39:05.760263 (+ 2989us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:05.760274 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:39:05.760347 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:39:05.760908 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:39:05.760914 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:39:05.760941 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:39:05.761092 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:05.761100 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:05.762696 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:05.762700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:05.764109 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:05.764120 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:05.765275 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:05.765279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:05.765396 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:05.765443 (+ 47us) client_negotiation.cc:769] Sending connection context
0619 14:39:05.765528 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 14:39:05.765643 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2936,"thread_start_us":77,"threads_started":1}
I20260619 14:39:05.766829 12136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:05.758017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55172 (local address 127.0.98.3:42381)
0619 14:39:05.759684 (+ 1667us) server_negotiation.cc:206] Beginning negotiation
0619 14:39:05.759696 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:39:05.760368 (+ 672us) server_negotiation.cc:407] Connection header received
0619 14:39:05.760459 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:39:05.760465 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:39:05.760687 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:39:05.761257 (+ 570us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:39:05.761399 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:05.762569 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:05.764292 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:05.765140 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:05.766413 (+ 1273us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:05.766558 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 14:39:05.766607 (+ 49us) server_negotiation.cc:299] Negotiation successful
0619 14:39:05.766693 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1563,"thread_start_us":106,"threads_started":1}
I20260619 14:39:05.770370 12054 tablet_service.cc:1467] Processing CreateTablet for tablet 9f451838ecec47aea78750f9f7af9ac3 (DEFAULT_TABLE table=test-table [id=18a1576e275c41b78a1b26568749bbec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:39:05.771651 12054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f451838ecec47aea78750f9f7af9ac3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:39:05.771659 12133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:05.759019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45227 (local address 127.0.0.1:34588)
0619 14:39:05.760492 (+ 1473us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:05.760502 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 14:39:05.760559 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:39:05.763901 (+ 3342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:39:05.763904 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:39:05.763923 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:39:05.764013 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:05.764019 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:05.769146 (+ 5127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:05.769149 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:05.770238 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:05.770248 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:05.771300 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:05.771303 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:05.771396 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:05.771430 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 14:39:05.771481 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 14:39:05.771539 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1428,"thread_start_us":78,"threads_started":1}
I20260619 14:39:05.772969 12138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:05.759388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34588 (local address 127.0.98.2:45227)
0619 14:39:05.761292 (+ 1904us) server_negotiation.cc:206] Beginning negotiation
0619 14:39:05.761307 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:39:05.761327 (+ 20us) server_negotiation.cc:407] Connection header received
0619 14:39:05.761390 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:39:05.761395 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:39:05.761578 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:39:05.764144 (+ 2566us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:39:05.764272 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:05.765426 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:05.770407 (+ 4981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:05.771174 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:05.772541 (+ 1367us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:05.772680 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 14:39:05.772746 (+ 66us) server_negotiation.cc:299] Negotiation successful
0619 14:39:05.772834 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1815,"thread_start_us":87,"threads_started":1}
I20260619 14:39:05.773591 12137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:05.757151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35781 (local address 127.0.0.1:33264)
0619 14:39:05.759910 (+ 2759us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:05.759960 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 14:39:05.760075 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:39:05.761724 (+ 1649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:39:05.761727 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:39:05.761743 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:39:05.761831 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:05.761836 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:05.765561 (+ 3725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:05.765565 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:05.766690 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:05.766700 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:05.773223 (+ 6523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:05.773226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:05.773325 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:05.773347 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 14:39:05.773394 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 14:39:05.773466 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2599,"thread_start_us":122,"threads_started":1}
I20260619 14:39:05.774385 12135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:05.757994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33264 (local address 127.0.98.1:35781)
0619 14:39:05.760781 (+ 2787us) server_negotiation.cc:206] Beginning negotiation
0619 14:39:05.760787 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:39:05.760808 (+ 21us) server_negotiation.cc:407] Connection header received
0619 14:39:05.760874 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:39:05.760879 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:39:05.761057 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:39:05.762592 (+ 1535us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:39:05.762720 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:05.763768 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:05.768033 (+ 4265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:05.769018 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:05.773968 (+ 4950us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:05.774104 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 14:39:05.774153 (+ 49us) server_negotiation.cc:299] Negotiation successful
0619 14:39:05.774253 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2671,"thread_start_us":96,"threads_started":1}
I20260619 14:39:05.777521 11913 tablet_service.cc:1467] Processing CreateTablet for tablet 9f451838ecec47aea78750f9f7af9ac3 (DEFAULT_TABLE table=test-table [id=18a1576e275c41b78a1b26568749bbec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:39:05.778622 11913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f451838ecec47aea78750f9f7af9ac3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:39:05.787360 12140 tablet_bootstrap.cc:492] T 9f451838ecec47aea78750f9f7af9ac3 P b9b391eb67904b468f986e432f881149: Bootstrap starting.
I20260619 14:39:05.787760 11767 tablet_service.cc:1467] Processing CreateTablet for tablet 9f451838ecec47aea78750f9f7af9ac3 (DEFAULT_TABLE table=test-table [id=18a1576e275c41b78a1b26568749bbec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:39:05.789458 11767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f451838ecec47aea78750f9f7af9ac3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:39:05.791107 12140 tablet_bootstrap.cc:654] T 9f451838ecec47aea78750f9f7af9ac3 P b9b391eb67904b468f986e432f881149: Neither blocks nor log segments found. Creating new log.
I20260619 14:39:05.791989 12140 log.cc:824] T 9f451838ecec47aea78750f9f7af9ac3 P b9b391eb67904b468f986e432f881149: Log is configured to *not* fsync() on all Append() calls
I20260619 14:39:05.793987 12140 tablet_bootstrap.cc:492] T 9f451838ecec47aea78750f9f7af9ac3 P b9b391eb67904b468f986e432f881149: No bootstrap required, opened a new log
I20260619 14:39:05.794194 12140 ts_tablet_manager.cc:1397] T 9f451838ecec47aea78750f9f7af9ac3 P b9b391eb67904b468f986e432f881149: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260619 14:39:05.799837 12140 raft_consensus.cc:348] T 9f451838ecec47aea78750f9f7af9ac3 P b9b391eb67904b468f986e432f881149 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e52d2479e3fe45f79121b006c283d330" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35781 } } peers { permanent_uuid: "73d82159532149399ffd45a28df5ef62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42381 } } peers { permanent_uuid: "b9b391eb67904b468f986e432f881149" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45227 } }
I20260619 14:39:05.800278 12140 raft_consensus.cc:374] T 9f451838ecec47aea78750f9f7af9ac3 P b9b391eb67904b468f986e432f881149 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:39:05.800371 12140 raft_consensus.cc:729] T 9f451838ecec47aea78750f9f7af9ac3 P b9b391eb67904b468f986e432f881149 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9b391eb67904b468f986e432f881149, State: Initialized, Role: FOLLOWER
I20260619 14:39:05.800984 12140 consensus_queue.cc:260] T 9f451838ecec47aea78750f9f7af9ac3 P b9b391eb67904b468f986e432f881149 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e52d2479e3fe45f79121b006c283d330" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35781 } } peers { permanent_uuid: "73d82159532149399ffd45a28df5ef62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42381 } } peers { permanent_uuid: "b9b391eb67904b468f986e432f881149" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45227 } }
I20260619 14:39:05.802109 12140 ts_tablet_manager.cc:1428] T 9f451838ecec47aea78750f9f7af9ac3 P b9b391eb67904b468f986e432f881149: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260619 14:39:05.802502 11987 heartbeater.cc:503] Master 127.0.98.62:36665 was elected leader, sending a full tablet report...
I20260619 14:39:05.806278 12143 tablet_bootstrap.cc:492] T 9f451838ecec47aea78750f9f7af9ac3 P e52d2479e3fe45f79121b006c283d330: Bootstrap starting.
I20260619 14:39:05.807668 12139 tablet_bootstrap.cc:492] T 9f451838ecec47aea78750f9f7af9ac3 P 73d82159532149399ffd45a28df5ef62: Bootstrap starting.
I20260619 14:39:05.810346 12139 tablet_bootstrap.cc:654] T 9f451838ecec47aea78750f9f7af9ac3 P 73d82159532149399ffd45a28df5ef62: Neither blocks nor log segments found. Creating new log.
I20260619 14:39:05.810827 12143 tablet_bootstrap.cc:654] T 9f451838ecec47aea78750f9f7af9ac3 P e52d2479e3fe45f79121b006c283d330: Neither blocks nor log segments found. Creating new log.
I20260619 14:39:05.811342 12139 log.cc:824] T 9f451838ecec47aea78750f9f7af9ac3 P 73d82159532149399ffd45a28df5ef62: Log is configured to *not* fsync() on all Append() calls
I20260619 14:39:05.811661 12143 log.cc:824] T 9f451838ecec47aea78750f9f7af9ac3 P e52d2479e3fe45f79121b006c283d330: Log is configured to *not* fsync() on all Append() calls
I20260619 14:39:05.813858 12143 tablet_bootstrap.cc:492] T 9f451838ecec47aea78750f9f7af9ac3 P e52d2479e3fe45f79121b006c283d330: No bootstrap required, opened a new log
I20260619 14:39:05.814057 12143 ts_tablet_manager.cc:1397] T 9f451838ecec47aea78750f9f7af9ac3 P e52d2479e3fe45f79121b006c283d330: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260619 14:39:05.815636 12139 tablet_bootstrap.cc:492] T 9f451838ecec47aea78750f9f7af9ac3 P 73d82159532149399ffd45a28df5ef62: No bootstrap required, opened a new log
I20260619 14:39:05.815891 12139 ts_tablet_manager.cc:1397] T 9f451838ecec47aea78750f9f7af9ac3 P 73d82159532149399ffd45a28df5ef62: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 14:39:05.818872 12143 raft_consensus.cc:348] T 9f451838ecec47aea78750f9f7af9ac3 P e52d2479e3fe45f79121b006c283d330 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e52d2479e3fe45f79121b006c283d330" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35781 } } peers { permanent_uuid: "73d82159532149399ffd45a28df5ef62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42381 } } peers { permanent_uuid: "b9b391eb67904b468f986e432f881149" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45227 } }
I20260619 14:39:05.820190 12143 raft_consensus.cc:374] T 9f451838ecec47aea78750f9f7af9ac3 P e52d2479e3fe45f79121b006c283d330 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:39:05.820278 12143 raft_consensus.cc:729] T 9f451838ecec47aea78750f9f7af9ac3 P e52d2479e3fe45f79121b006c283d330 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e52d2479e3fe45f79121b006c283d330, State: Initialized, Role: FOLLOWER
I20260619 14:39:05.820825 12143 consensus_queue.cc:260] T 9f451838ecec47aea78750f9f7af9ac3 P e52d2479e3fe45f79121b006c283d330 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e52d2479e3fe45f79121b006c283d330" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35781 } } peers { permanent_uuid: "73d82159532149399ffd45a28df5ef62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42381 } } peers { permanent_uuid: "b9b391eb67904b468f986e432f881149" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45227 } }
I20260619 14:39:05.821936 12143 ts_tablet_manager.cc:1428] T 9f451838ecec47aea78750f9f7af9ac3 P e52d2479e3fe45f79121b006c283d330: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260619 14:39:05.822307 11851 heartbeater.cc:503] Master 127.0.98.62:36665 was elected leader, sending a full tablet report...
I20260619 14:39:05.825032 12139 raft_consensus.cc:348] T 9f451838ecec47aea78750f9f7af9ac3 P 73d82159532149399ffd45a28df5ef62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e52d2479e3fe45f79121b006c283d330" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35781 } } peers { permanent_uuid: "73d82159532149399ffd45a28df5ef62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42381 } } peers { permanent_uuid: "b9b391eb67904b468f986e432f881149" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45227 } }
I20260619 14:39:05.826053 12139 raft_consensus.cc:374] T 9f451838ecec47aea78750f9f7af9ac3 P 73d82159532149399ffd45a28df5ef62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:39:05.826170 12139 raft_consensus.cc:729] T 9f451838ecec47aea78750f9f7af9ac3 P 73d82159532149399ffd45a28df5ef62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73d82159532149399ffd45a28df5ef62, State: Initialized, Role: FOLLOWER
I20260619 14:39:05.826712 12139 consensus_queue.cc:260] T 9f451838ecec47aea78750f9f7af9ac3 P 73d82159532149399ffd45a28df5ef62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e52d2479e3fe45f79121b006c283d330" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35781 } } peers { permanent_uuid: "73d82159532149399ffd45a28df5ef62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42381 } } peers { permanent_uuid: "b9b391eb67904b468f986e432f881149" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45227 } }
I20260619 14:39:05.827776 12139 ts_tablet_manager.cc:1428] T 9f451838ecec47aea78750f9f7af9ac3 P 73d82159532149399ffd45a28df5ef62: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20260619 14:39:05.828158 12121 heartbeater.cc:503] Master 127.0.98.62:36665 was elected leader, sending a full tablet report...
Jun 19 14:39:05 dist-test-slave-32hj krb5kdc[11161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879944, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260619 14:39:05.945078 12122 tablet.cc:2369] T 9f451838ecec47aea78750f9f7af9ac3 P 73d82159532149399ffd45a28df5ef62: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:39:05.963408 11852 tablet.cc:2369] T 9f451838ecec47aea78750f9f7af9ac3 P e52d2479e3fe45f79121b006c283d330: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:39:06.013747 12142 raft_consensus.cc:482] T 9f451838ecec47aea78750f9f7af9ac3 P b9b391eb67904b468f986e432f881149 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:39:06.014021 12142 raft_consensus.cc:504] T 9f451838ecec47aea78750f9f7af9ac3 P b9b391eb67904b468f986e432f881149 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e52d2479e3fe45f79121b006c283d330" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35781 } } peers { permanent_uuid: "73d82159532149399ffd45a28df5ef62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42381 } } peers { permanent_uuid: "b9b391eb67904b468f986e432f881149" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45227 } }
I20260619 14:39:06.015516 12142 leader_election.cc:288] T 9f451838ecec47aea78750f9f7af9ac3 P b9b391eb67904b468f986e432f881149 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e52d2479e3fe45f79121b006c283d330 (127.0.98.1:35781), 73d82159532149399ffd45a28df5ef62 (127.0.98.3:42381)
I20260619 14:39:06.021467 12135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:06.016101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60833 (local address 127.0.98.1:35781)
0619 14:39:06.016232 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 14:39:06.016237 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:39:06.016371 (+ 134us) server_negotiation.cc:407] Connection header received
0619 14:39:06.016568 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:39:06.016571 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:39:06.016620 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:39:06.016664 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:39:06.017091 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:06.017919 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:06.019790 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:06.020651 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:06.020828 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:06.020900 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 14:39:06.021233 (+ 333us) server_negotiation.cc:299] Negotiation successful
0619 14:39:06.021318 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 14:39:06.021636 12136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:06.016266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38257 (local address 127.0.98.3:42381)
0619 14:39:06.016367 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 14:39:06.016372 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:39:06.016569 (+ 197us) server_negotiation.cc:407] Connection header received
0619 14:39:06.016805 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:39:06.016808 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:39:06.016855 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:39:06.016903 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:39:06.017288 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:06.018084 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:06.020293 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:06.020924 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:06.021003 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:06.021064 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 14:39:06.021413 (+ 349us) server_negotiation.cc:299] Negotiation successful
0619 14:39:06.021515 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 14:39:06.022123 12151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:06.015756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35781 (local address 127.0.98.2:60833)
0619 14:39:06.016282 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:06.016297 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:39:06.016451 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:39:06.016833 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:39:06.016836 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:39:06.016852 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:39:06.016976 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:06.016982 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:06.018258 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:06.018261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:06.019570 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:06.019581 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:06.021001 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:06.021004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:06.021099 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:06.021122 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 14:39:06.021890 (+ 768us) client_negotiation.cc:240] Negotiation successful
0619 14:39:06.021972 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"mutex_wait_us":2,"thread_start_us":67,"threads_started":1}
I20260619 14:39:06.022389 12150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:06.015865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42381 (local address 127.0.98.2:38257)
0619 14:39:06.016493 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:06.016506 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:39:06.016635 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:39:06.017039 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:39:06.017042 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:39:06.017059 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:39:06.017167 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:06.017173 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:06.018258 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:06.018260 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:06.020113 (+ 1853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:06.020125 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:06.021157 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:06.021161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:06.021261 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:06.021285 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 14:39:06.022011 (+ 726us) client_negotiation.cc:240] Negotiation successful
0619 14:39:06.022090 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"mutex_wait_us":84,"thread_start_us":117,"threads_started":1}
I20260619 14:39:06.022891 11805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f451838ecec47aea78750f9f7af9ac3" candidate_uuid: "b9b391eb67904b468f986e432f881149" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e52d2479e3fe45f79121b006c283d330" is_pre_election: true
I20260619 14:39:06.023226 11805 raft_consensus.cc:2454] T 9f451838ecec47aea78750f9f7af9ac3 P e52d2479e3fe45f79121b006c283d330 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b391eb67904b468f986e432f881149 in term 0.
I20260619 14:39:06.023232 12059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f451838ecec47aea78750f9f7af9ac3" candidate_uuid: "b9b391eb67904b468f986e432f881149" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73d82159532149399ffd45a28df5ef62" is_pre_election: true
I20260619 14:39:06.023492 12059 raft_consensus.cc:2454] T 9f451838ecec47aea78750f9f7af9ac3 P 73d82159532149399ffd45a28df5ef62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b391eb67904b468f986e432f881149 in term 0.
I20260619 14:39:06.023772 11877 leader_election.cc:302] T 9f451838ecec47aea78750f9f7af9ac3 P b9b391eb67904b468f986e432f881149 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9b391eb67904b468f986e432f881149, e52d2479e3fe45f79121b006c283d330; no voters:
I20260619 14:39:06.024185 12142 raft_consensus.cc:2781] T 9f451838ecec47aea78750f9f7af9ac3 P b9b391eb67904b468f986e432f881149 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:39:06.024281 12142 raft_consensus.cc:482] T 9f451838ecec47aea78750f9f7af9ac3 P b9b391eb67904b468f986e432f881149 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:39:06.024333 12142 raft_consensus.cc:3037] T 9f451838ecec47aea78750f9f7af9ac3 P b9b391eb67904b468f986e432f881149 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:39:06.025632 12142 raft_consensus.cc:504] T 9f451838ecec47aea78750f9f7af9ac3 P b9b391eb67904b468f986e432f881149 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e52d2479e3fe45f79121b006c283d330" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35781 } } peers { permanent_uuid: "73d82159532149399ffd45a28df5ef62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42381 } } peers { permanent_uuid: "b9b391eb67904b468f986e432f881149" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45227 } }
I20260619 14:39:06.026003 12142 leader_election.cc:288] T 9f451838ecec47aea78750f9f7af9ac3 P b9b391eb67904b468f986e432f881149 [CANDIDATE]: Term 1 election: Requested vote from peers e52d2479e3fe45f79121b006c283d330 (127.0.98.1:35781), 73d82159532149399ffd45a28df5ef62 (127.0.98.3:42381)
I20260619 14:39:06.026373 11805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f451838ecec47aea78750f9f7af9ac3" candidate_uuid: "b9b391eb67904b468f986e432f881149" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e52d2479e3fe45f79121b006c283d330"
I20260619 14:39:06.026425 12059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f451838ecec47aea78750f9f7af9ac3" candidate_uuid: "b9b391eb67904b468f986e432f881149" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73d82159532149399ffd45a28df5ef62"
I20260619 14:39:06.026494 11805 raft_consensus.cc:3037] T 9f451838ecec47aea78750f9f7af9ac3 P e52d2479e3fe45f79121b006c283d330 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:39:06.026541 12059 raft_consensus.cc:3037] T 9f451838ecec47aea78750f9f7af9ac3 P 73d82159532149399ffd45a28df5ef62 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:39:06.027752 12059 raft_consensus.cc:2454] T 9f451838ecec47aea78750f9f7af9ac3 P 73d82159532149399ffd45a28df5ef62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b391eb67904b468f986e432f881149 in term 1.
I20260619 14:39:06.027773 11805 raft_consensus.cc:2454] T 9f451838ecec47aea78750f9f7af9ac3 P e52d2479e3fe45f79121b006c283d330 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b391eb67904b468f986e432f881149 in term 1.
I20260619 14:39:06.028055 11874 leader_election.cc:302] T 9f451838ecec47aea78750f9f7af9ac3 P b9b391eb67904b468f986e432f881149 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73d82159532149399ffd45a28df5ef62, b9b391eb67904b468f986e432f881149; no voters:
I20260619 14:39:06.028235 12142 raft_consensus.cc:2781] T 9f451838ecec47aea78750f9f7af9ac3 P b9b391eb67904b468f986e432f881149 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:39:06.028458 12142 raft_consensus.cc:686] T 9f451838ecec47aea78750f9f7af9ac3 P b9b391eb67904b468f986e432f881149 [term 1 LEADER]: Becoming Leader. State: Replica: b9b391eb67904b468f986e432f881149, State: Running, Role: LEADER
I20260619 14:39:06.029084 12142 consensus_queue.cc:237] T 9f451838ecec47aea78750f9f7af9ac3 P b9b391eb67904b468f986e432f881149 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e52d2479e3fe45f79121b006c283d330" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35781 } } peers { permanent_uuid: "73d82159532149399ffd45a28df5ef62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42381 } } peers { permanent_uuid: "b9b391eb67904b468f986e432f881149" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45227 } }
I20260619 14:39:06.032667 11637 catalog_manager.cc:5495] T 9f451838ecec47aea78750f9f7af9ac3 P b9b391eb67904b468f986e432f881149 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9b391eb67904b468f986e432f881149 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "b9b391eb67904b468f986e432f881149" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e52d2479e3fe45f79121b006c283d330" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73d82159532149399ffd45a28df5ef62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b391eb67904b468f986e432f881149" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45227 } health_report { overall_health: HEALTHY } } }
W20260619 14:39:06.039199 11988 tablet.cc:2369] T 9f451838ecec47aea78750f9f7af9ac3 P b9b391eb67904b468f986e432f881149: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:39:06.056288 12138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:06.052492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34594 (local address 127.0.98.2:45227)
0619 14:39:06.052630 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 14:39:06.052635 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:39:06.052657 (+ 22us) server_negotiation.cc:407] Connection header received
0619 14:39:06.052848 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:39:06.052851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:39:06.052918 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:39:06.053005 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:39:06.053240 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:06.053728 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:06.054562 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:06.054917 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:06.055043 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:06.055445 (+ 402us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:39:06.055876 (+ 431us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:39:06.055992 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 14:39:06.056083 (+ 91us) server_negotiation.cc:299] Negotiation successful
0619 14:39:06.056140 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 14:39:06.062443 12059 raft_consensus.cc:1261] T 9f451838ecec47aea78750f9f7af9ac3 P 73d82159532149399ffd45a28df5ef62 [term 1 FOLLOWER]: Refusing update from remote peer b9b391eb67904b468f986e432f881149: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:39:06.063212 12152 consensus_queue.cc:1035] T 9f451838ecec47aea78750f9f7af9ac3 P b9b391eb67904b468f986e432f881149 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73d82159532149399ffd45a28df5ef62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:39:06.063324 11805 raft_consensus.cc:1261] T 9f451838ecec47aea78750f9f7af9ac3 P e52d2479e3fe45f79121b006c283d330 [term 1 FOLLOWER]: Refusing update from remote peer b9b391eb67904b468f986e432f881149: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:39:06.063874 12152 consensus_queue.cc:1035] T 9f451838ecec47aea78750f9f7af9ac3 P b9b391eb67904b468f986e432f881149 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e52d2479e3fe45f79121b006c283d330" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:39:06.075078 12164 mvcc.cc:205] Tried to move back new op lower bound from 7298580259064406016 to 7298580258938421248. Current Snapshot: MvccSnapshot[applied={T|T < 7298580259064406016}]
I20260619 14:39:06.081043 11636 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58278:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:39:06.081210 11636 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58278:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:39:06.083841 11636 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9fa9c20067164da7b786530e76b84604: Sending DeleteTablet for 3 replicas of tablet 9f451838ecec47aea78750f9f7af9ac3
I20260619 14:39:06.084558 12054 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f451838ecec47aea78750f9f7af9ac3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:39:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55172
I20260619 14:39:06.084622 11913 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f451838ecec47aea78750f9f7af9ac3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:39:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34588
I20260619 14:39:06.084815 11767 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f451838ecec47aea78750f9f7af9ac3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:39:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33264
I20260619 14:39:06.085078 12171 tablet_replica.cc:330] T 9f451838ecec47aea78750f9f7af9ac3 P 73d82159532149399ffd45a28df5ef62: stopping tablet replica
I20260619 14:39:06.085238 12173 tablet_replica.cc:330] T 9f451838ecec47aea78750f9f7af9ac3 P e52d2479e3fe45f79121b006c283d330: stopping tablet replica
I20260619 14:39:06.085391 12171 raft_consensus.cc:2229] T 9f451838ecec47aea78750f9f7af9ac3 P 73d82159532149399ffd45a28df5ef62 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:39:06.085521 12173 raft_consensus.cc:2229] T 9f451838ecec47aea78750f9f7af9ac3 P e52d2479e3fe45f79121b006c283d330 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:39:06.085654 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 11718
I20260619 14:39:06.085692 12171 raft_consensus.cc:2258] T 9f451838ecec47aea78750f9f7af9ac3 P 73d82159532149399ffd45a28df5ef62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:39:06.086002 12172 tablet_replica.cc:330] T 9f451838ecec47aea78750f9f7af9ac3 P b9b391eb67904b468f986e432f881149: stopping tablet replica
I20260619 14:39:06.086339 12172 raft_consensus.cc:2229] T 9f451838ecec47aea78750f9f7af9ac3 P b9b391eb67904b468f986e432f881149 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:39:06.086683 12172 raft_consensus.cc:2258] T 9f451838ecec47aea78750f9f7af9ac3 P b9b391eb67904b468f986e432f881149 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:39:06.090243 12172 ts_tablet_manager.cc:1905] T 9f451838ecec47aea78750f9f7af9ac3 P b9b391eb67904b468f986e432f881149: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:39:06.093071 12172 ts_tablet_manager.cc:1918] T 9f451838ecec47aea78750f9f7af9ac3 P b9b391eb67904b468f986e432f881149: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:39:06.093194 12172 log.cc:1196] T 9f451838ecec47aea78750f9f7af9ac3 P b9b391eb67904b468f986e432f881149: Deleting WAL directory at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ts-1/wal/wals/9f451838ecec47aea78750f9f7af9ac3
I20260619 14:39:06.093525 12172 ts_tablet_manager.cc:1939] T 9f451838ecec47aea78750f9f7af9ac3 P b9b391eb67904b468f986e432f881149: Deleting consensus metadata
W20260619 14:39:06.093536 11624 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35781 (error 108)
I20260619 14:39:06.094195 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 11857
I20260619 14:39:06.095067 12171 ts_tablet_manager.cc:1905] T 9f451838ecec47aea78750f9f7af9ac3 P 73d82159532149399ffd45a28df5ef62: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:39:06.096678 12137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:06.096373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35781 (local address 127.0.0.1:33268)
0619 14:39:06.096508 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:06.096583 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 14:39:06.096985 11624 catalog_manager.cc:4584] TS e52d2479e3fe45f79121b006c283d330 (127.0.98.1:35781): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9f451838ecec47aea78750f9f7af9ac3: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35781: connect: Connection refused (error 111)
I20260619 14:39:06.097539 12171 ts_tablet_manager.cc:1918] T 9f451838ecec47aea78750f9f7af9ac3 P 73d82159532149399ffd45a28df5ef62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:39:06.097635 12171 log.cc:1196] T 9f451838ecec47aea78750f9f7af9ac3 P 73d82159532149399ffd45a28df5ef62: Deleting WAL directory at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781879837648036-392-0/minicluster-data/ts-2/wal/wals/9f451838ecec47aea78750f9f7af9ac3
I20260619 14:39:06.097954 12171 ts_tablet_manager.cc:1939] T 9f451838ecec47aea78750f9f7af9ac3 P 73d82159532149399ffd45a28df5ef62: Deleting consensus metadata
I20260619 14:39:06.099184 11621 catalog_manager.cc:4841] TS 73d82159532149399ffd45a28df5ef62 (127.0.98.3:42381): tablet 9f451838ecec47aea78750f9f7af9ac3 (table test-table [id=18a1576e275c41b78a1b26568749bbec]) successfully deleted
I20260619 14:39:06.100420 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 11992
I20260619 14:39:06.101737 12137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:06.100540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45227 (local address 127.0.0.1:34604)
0619 14:39:06.101621 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:06.101669 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:45227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1008}
I20260619 14:39:06.105373 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 11605
I20260619 14:39:06.112460 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 14:39:06.446632 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 14:39:06.446764 392 mini_ranger.cc:66] Stopping Ranger...
I20260619 14:39:06.781046 392 mini_ranger.cc:68] Stopped Ranger
2026-06-19 14:39:06.781 UTC [11191] LOG: received smart shutdown request
2026-06-19 14:39:06.784 UTC [11191] LOG: background worker "logical replication launcher" (PID 11200) exited with exit code 1
2026-06-19 14:39:06.784 UTC [11195] LOG: shutting down
2026-06-19 14:39:06.917 UTC [11191] LOG: database system is shut down
2026-06-19T14:39:06Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45087 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:39:07 dist-test-slave-32hj krb5kdc[12183](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:39:07 dist-test-slave-32hj krb5kdc[12183](info): set up 2 sockets
Jun 19 14:39:07 dist-test-slave-32hj krb5kdc[12183](info): commencing operation
krb5kdc: starting...
W20260619 14:39:07.968485 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/krb5kdc/test-user.keytab.
Jun 19 14:39:08 dist-test-slave-32hj krb5kdc[12183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:39:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:39:08Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260619 14:39:08.121353 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task7PHJY_/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/postgres -l logfile start
2026-06-19 14:39:10.370 UTC [12213] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 14:39:10.370 UTC [12213] LOG: listening on IPv4 address "127.0.98.20", port 54501
2026-06-19 14:39:10.373 UTC [12213] LOG: listening on Unix socket "/tmp/.s.PGSQL.54501"
2026-06-19 14:39:10.392 UTC [12216] LOG: database system was shut down at 2026-06-19 14:39:09 UTC
2026-06-19 14:39:10.395 UTC [12213] LOG: database system is ready to accept connections
I20260619 14:39:11.268682 392 mini_postgres.cc:96] Postgres bound to 54501
2026-06-19 14:39:11.274 UTC [12224] FATAL: database "slave" does not exist
127.0.98.20:54501 - accepting connections
I20260619 14:39:11.275172 392 mini_ranger.cc:161] Starting Ranger...
I20260619 14:39:11.289229 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 14:39:11.707127 392 mini_ranger.cc:87] Created ranger Postgres database
I20260619 14:39:11.707211 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ranger-admin
2026-06-19 14:39:11,978 [I] DB FLAVOR :POSTGRES
2026-06-19 14:39:11,979 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 14:39:11,979 [I] Checking connection..
2026-06-19 14:39:11,979 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7PHJY_/build/debug/bin/postgresql.jar:/tmp/dist-test-task7PHJY_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 14:39:12,232 [I] Checking connection passed.
2026-06-19 14:39:12,232 [I] --------- Verifying version history table ---------
2026-06-19 14:39:12,233 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7PHJY_/build/debug/bin/postgresql.jar:/tmp/dist-test-task7PHJY_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 14:39:12,455 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 14:39:12,456 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7PHJY_/build/debug/bin/postgresql.jar:/tmp/dist-test-task7PHJY_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 14:39:12,675 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 14:39:12,676 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 14:39:12,676 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7PHJY_/build/debug/bin/postgresql.jar:/tmp/dist-test-task7PHJY_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54501/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7PHJY_/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 14:39:12.923 UTC [12288] WARNING: there is no transaction in progress
2026-06-19 14:39:12,929 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 14:39:12,929 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7PHJY_/build/debug/bin/postgresql.jar:/tmp/dist-test-task7PHJY_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 14:39:13,181 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 14:39:13,181 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 14:39:13,182 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7PHJY_/build/debug/bin/postgresql.jar:/tmp/dist-test-task7PHJY_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 14:39:13,400 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7PHJY_/build/debug/bin/postgresql.jar:/tmp/dist-test-task7PHJY_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 14:39:13,648 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7PHJY_/build/debug/bin/postgresql.jar:/tmp/dist-test-task7PHJY_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-32hj.c.gcp-upstream.internal','N') ;"
2026-06-19 14:39:13,882 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7PHJY_/build/debug/bin/postgresql.jar:/tmp/dist-test-task7PHJY_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 14:39:14,117 [I] Table x_portal_user does not exist in database ranger
2026-06-19 14:39:14,118 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7PHJY_/build/debug/bin/postgresql.jar:/tmp/dist-test-task7PHJY_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 14:39:14,357 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 14:39:14,357 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7PHJY_/build/debug/bin/postgresql.jar:/tmp/dist-test-task7PHJY_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 14:39:14,596 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7PHJY_/build/debug/bin/postgresql.jar:/tmp/dist-test-task7PHJY_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 14:39:14,827 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 14:39:14,828 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7PHJY_/build/debug/bin/postgresql.jar:/tmp/dist-test-task7PHJY_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54501/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7PHJY_/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 14:39:15.723 UTC [12396] WARNING: there is no transaction in progress
2026-06-19 14:39:15.738 UTC [12396] WARNING: there is no transaction in progress
2026-06-19 14:39:15.753 UTC [12396] WARNING: there is no transaction in progress
2026-06-19 14:39:15.767 UTC [12396] WARNING: there is no transaction in progress
2026-06-19 14:39:15.782 UTC [12396] WARNING: there is no transaction in progress
2026-06-19 14:39:15.799 UTC [12396] WARNING: there is no transaction in progress
2026-06-19 14:39:15.862 UTC [12396] WARNING: there is no transaction in progress
2026-06-19 14:39:15.868 UTC [12396] WARNING: there is no transaction in progress
2026-06-19 14:39:15.881 UTC [12396] WARNING: there is no transaction in progress
2026-06-19 14:39:15.890 UTC [12396] WARNING: there is no transaction in progress
2026-06-19 14:39:15.899 UTC [12396] WARNING: there is no transaction in progress
2026-06-19 14:39:15.912 UTC [12396] WARNING: there is no transaction in progress
2026-06-19 14:39:15.921 UTC [12396] WARNING: there is no transaction in progress
2026-06-19 14:39:15.926 UTC [12396] WARNING: there is no transaction in progress
2026-06-19 14:39:15.932 UTC [12396] WARNING: there is no transaction in progress
2026-06-19 14:39:15.937 UTC [12396] WARNING: there is no transaction in progress
2026-06-19 14:39:16,489 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 14:39:16,490 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7PHJY_/build/debug/bin/postgresql.jar:/tmp/dist-test-task7PHJY_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 14:39:16,717 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 14:39:16,717 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7PHJY_/build/debug/bin/postgresql.jar:/tmp/dist-test-task7PHJY_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 14:39:16,938 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 14:39:16,939 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7PHJY_/build/debug/bin/postgresql.jar:/tmp/dist-test-task7PHJY_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 14:39:17,187 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 14:39:17,188 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7PHJY_/build/debug/bin/postgresql.jar:/tmp/dist-test-task7PHJY_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 14:39:17,414 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7PHJY_/build/debug/bin/postgresql.jar:/tmp/dist-test-task7PHJY_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 14:39:17,661 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7PHJY_/build/debug/bin/postgresql.jar:/tmp/dist-test-task7PHJY_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-32hj.c.gcp-upstream.internal';"
2026-06-19 14:39:17,894 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 14:39:17,894 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7PHJY_/build/debug/bin/postgresql.jar:/tmp/dist-test-task7PHJY_/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54501/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 14:39:18,138 [I] DB_PATCHES have already been applied
I20260619 14:39:18.143719 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ranger-admin:/tmp/dist-test-task7PHJY_/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task7PHJY_/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task7PHJY_/build/debug/bin/postgresql.jar:/tmp/dist-test-task7PHJY_/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 14:39:18.143781 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260619 14:39:18.146870 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:47571
Jun 19, 2026 2:39:18 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 2:39:18 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 2:39:18 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 2:39:18 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 2:39:18 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 2:39:18 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 2:39:18 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 14:39:19.158982 392 mini_ranger.cc:160] Time spent starting Ranger: real 7.884s user 0.000s sys 0.008s
Jun 19, 2026 2:39:37 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 2:39:37 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Jun 19, 2026 2:39:37 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 2:39:37 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 2:39:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 2:39:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 2:39:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 2:39:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 2:39:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 2:39:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 2:39:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 2:39:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 2:39:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 2:39:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 2:39:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 2:39:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 2:39:37 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 2:39:38 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 14:39:39.250909 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 14:39:39 dist-test-slave-32hj krb5kdc[12183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879979, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 14:39:39.334362 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 14:39:39.342517 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 14:39:40.058470 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 14:39:40.058549 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ranger-kms
I20260619 14:39:40.058565 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:54501
2026-06-19 14:39:40,210 [I] DB FLAVOR :POSTGRES
2026-06-19 14:39:40,211 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 14:39:40,211 [I] Checking connection
2026-06-19 14:39:40,211 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7PHJY_/build/debug/bin/postgresql.jar:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54501/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 14:39:40,452 [I] connection success
2026-06-19 14:39:40,452 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 14:39:40,452 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 14:39:40,452 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7PHJY_/build/debug/bin/postgresql.jar:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54501/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 14:39:40,680 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 14:39:40,680 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 14:39:40,680 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 14:39:40,680 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7PHJY_/build/debug/bin/postgresql.jar:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54501/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 14:39:40,947 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 14:39:41.367867 392 mini_ranger_kms.cc:310] Created kms service
I20260619 14:39:41.538919 392 mini_ranger_kms.cc:326] Created kudu user
I20260619 14:39:41.581367 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 14:39:41 dist-test-slave-32hj krb5kdc[12183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879979, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 14:39:41.962944 392 mini_ranger_kms.cc:384] Added ranger policy
I20260619 14:39:41.963074 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ranger-kms:/tmp/dist-test-task7PHJY_/build/debug/bin/postgresql.jar:/tmp/dist-test-task7PHJY_/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task7PHJY_/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 14:39:41.963092 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260619 14:39:41.966502 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12576
I20260619 14:39:41.966594 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:45515
Jun 19, 2026 2:39:42 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 2:39:42 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 2:39:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 2:39:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 2:39:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 2:39:43 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 14:39:43.980451 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.646s user 0.000s sys 0.136s
I20260619 14:39:43.980587 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 14:39:43.980666 392 mini_ranger_kms.cc:401] 127.0.98.20:45515/kms/v1/keys
Jun 19 14:39:43 dist-test-slave-32hj krb5kdc[12183](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781879983, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 14:39:44 dist-test-slave-32hj krb5kdc[12183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879983, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 19 14:39:44 dist-test-slave-32hj krb5kdc[12183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879983, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 19 14:39:46 dist-test-slave-32hj krb5kdc[12183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:39:46.879557 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39091
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39091
--ranger_config_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:45515/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:39:46.986824 12627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:39:46.987082 12627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:39:46.987135 12627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:39:46.991104 12627 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:39:46.991158 12627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:39:46.991184 12627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:39:46.991205 12627 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:39:46.991230 12627 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:39:46.998351 12627 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:45515/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39091
--ranger_config_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:39:46.999588 12627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:39:47.000746 12627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:39:47.009234 12632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:39:47.009261 12633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:39:47.009306 12635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:39:47.009248 12627 server_base.cc:1034] running on GCE node
I20260619 14:39:47.010054 12627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:39:47.011190 12627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:39:47.012387 12627 hybrid_clock.cc:648] HybridClock initialized: now 1781879987012356 us; error 43 us; skew 500 ppm
Jun 19 14:39:47 dist-test-slave-32hj krb5kdc[12183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879987, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:39:47.019651 12627 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:39:47.020582 12627 webserver.cc:460] Webserver started at http://127.0.98.62:39007/ using document root <none> and password file <none>
I20260619 14:39:47.021166 12627 fs_manager.cc:359] Metadata directory not provided
I20260619 14:39:47.021229 12627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:39:47.021423 12627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 14:39:47 dist-test-slave-32hj krb5kdc[12183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879987, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 14:39:47.074632 12627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/master-0/data/instance:
uuid: "08cd4a52f1ad4e4cb0c75d4c328ff80d"
format_stamp: "Formatted at 2026-06-19 14:39:47 on dist-test-slave-32hj"
server_key: "ea4940c1d03f42092d95b24f8af0a361"
server_key_iv: "2301a9363fdd5b25c967505c528be288"
server_key_version: "kuduclusterkey@0"
I20260619 14:39:47.075237 12627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/master-0/wal/instance:
uuid: "08cd4a52f1ad4e4cb0c75d4c328ff80d"
format_stamp: "Formatted at 2026-06-19 14:39:47 on dist-test-slave-32hj"
server_key: "ea4940c1d03f42092d95b24f8af0a361"
server_key_iv: "2301a9363fdd5b25c967505c528be288"
server_key_version: "kuduclusterkey@0"
I20260619 14:39:47.079077 12627 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:39:47.095367 12642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:39:47.096700 12627 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:39:47.096890 12627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/master-0/wal
uuid: "08cd4a52f1ad4e4cb0c75d4c328ff80d"
format_stamp: "Formatted at 2026-06-19 14:39:47 on dist-test-slave-32hj"
server_key: "ea4940c1d03f42092d95b24f8af0a361"
server_key_iv: "2301a9363fdd5b25c967505c528be288"
server_key_version: "kuduclusterkey@0"
I20260619 14:39:47.097007 12627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:39:47.115818 12627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:39:47.118248 12627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:39:47.118420 12627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:39:47.125270 12627 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39091
I20260619 14:39:47.125331 12694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39091 every 8 connection(s)
I20260619 14:39:47.126190 12627 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:39:47.128829 12695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:39:47.134085 12695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08cd4a52f1ad4e4cb0c75d4c328ff80d: Bootstrap starting.
I20260619 14:39:47.136317 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 12627
I20260619 14:39:47.136431 12695 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08cd4a52f1ad4e4cb0c75d4c328ff80d: Neither blocks nor log segments found. Creating new log.
I20260619 14:39:47.136447 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/master-0/wal/instance
I20260619 14:39:47.136799 392 external_mini_cluster.cc:1428] Setting key c0636aebfa15682307bf9865a0da894b
I20260619 14:39:47.137275 12695 log.cc:824] T 00000000000000000000000000000000 P 08cd4a52f1ad4e4cb0c75d4c328ff80d: Log is configured to *not* fsync() on all Append() calls
I20260619 14:39:47.139122 12695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08cd4a52f1ad4e4cb0c75d4c328ff80d: No bootstrap required, opened a new log
I20260619 14:39:47.143085 12695 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08cd4a52f1ad4e4cb0c75d4c328ff80d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08cd4a52f1ad4e4cb0c75d4c328ff80d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39091 } }
I20260619 14:39:47.143353 12695 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08cd4a52f1ad4e4cb0c75d4c328ff80d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:39:47.143419 12695 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08cd4a52f1ad4e4cb0c75d4c328ff80d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08cd4a52f1ad4e4cb0c75d4c328ff80d, State: Initialized, Role: FOLLOWER
I20260619 14:39:47.143889 12695 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08cd4a52f1ad4e4cb0c75d4c328ff80d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08cd4a52f1ad4e4cb0c75d4c328ff80d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39091 } }
I20260619 14:39:47.144043 12695 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08cd4a52f1ad4e4cb0c75d4c328ff80d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:39:47.144109 12695 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08cd4a52f1ad4e4cb0c75d4c328ff80d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:39:47.144207 12695 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08cd4a52f1ad4e4cb0c75d4c328ff80d [term 0 FOLLOWER]: Advancing to term 1
Jun 19 14:39:47 dist-test-slave-32hj krb5kdc[12183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:39:47.145176 12695 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08cd4a52f1ad4e4cb0c75d4c328ff80d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08cd4a52f1ad4e4cb0c75d4c328ff80d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39091 } }
I20260619 14:39:47.145502 12695 leader_election.cc:302] T 00000000000000000000000000000000 P 08cd4a52f1ad4e4cb0c75d4c328ff80d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08cd4a52f1ad4e4cb0c75d4c328ff80d; no voters:
I20260619 14:39:47.145838 12695 leader_election.cc:288] T 00000000000000000000000000000000 P 08cd4a52f1ad4e4cb0c75d4c328ff80d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:39:47.145928 12700 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08cd4a52f1ad4e4cb0c75d4c328ff80d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:39:47.146166 12700 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08cd4a52f1ad4e4cb0c75d4c328ff80d [term 1 LEADER]: Becoming Leader. State: Replica: 08cd4a52f1ad4e4cb0c75d4c328ff80d, State: Running, Role: LEADER
I20260619 14:39:47.146799 12700 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08cd4a52f1ad4e4cb0c75d4c328ff80d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08cd4a52f1ad4e4cb0c75d4c328ff80d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39091 } }
I20260619 14:39:47.146998 12695 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08cd4a52f1ad4e4cb0c75d4c328ff80d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:39:47.148757 12701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08cd4a52f1ad4e4cb0c75d4c328ff80d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08cd4a52f1ad4e4cb0c75d4c328ff80d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08cd4a52f1ad4e4cb0c75d4c328ff80d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39091 } } }
I20260619 14:39:47.148886 12701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08cd4a52f1ad4e4cb0c75d4c328ff80d [sys.catalog]: This master's current role is: LEADER
I20260619 14:39:47.148980 12695 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 14:39:47.149147 12702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08cd4a52f1ad4e4cb0c75d4c328ff80d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08cd4a52f1ad4e4cb0c75d4c328ff80d. Latest consensus state: current_term: 1 leader_uuid: "08cd4a52f1ad4e4cb0c75d4c328ff80d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08cd4a52f1ad4e4cb0c75d4c328ff80d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39091 } } }
I20260619 14:39:47.149237 12702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08cd4a52f1ad4e4cb0c75d4c328ff80d [sys.catalog]: This master's current role is: LEADER
I20260619 14:39:47.152453 12714 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:39:47.152962 12699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:47.137920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57698 (local address 127.0.98.62:39091)
0619 14:39:47.138269 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0619 14:39:47.138284 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:39:47.138319 (+ 35us) server_negotiation.cc:407] Connection header received
0619 14:39:47.138908 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:39:47.138925 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:39:47.139249 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:39:47.139538 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:39:47.140383 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:47.141378 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:47.142527 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:47.142969 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:47.143086 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:47.145887 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:39:47.145894 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:39:47.145911 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:39:47.145943 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:39:47.147870 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:47.148271 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:47.148277 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:47.148283 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:47.148411 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:47.148748 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:47.148751 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:47.148753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:47.149279 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:39:47.149478 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0619 14:39:47.149878 (+ 400us) server_negotiation.cc:299] Negotiation successful
0619 14:39:47.150152 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":82,"threads_started":1}
I20260619 14:39:47.156154 12714 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:39:47.161497 12714 catalog_manager.cc:1347] Generated new cluster ID: fbdb41c28eda432e95af34becf4a1294
I20260619 14:39:47.161551 12714 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:39:47.174466 12714 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:39:47.175685 12714 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:39:47.189486 12714 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08cd4a52f1ad4e4cb0c75d4c328ff80d: Generated new TSK 0
I20260619 14:39:47.190265 12714 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 14:39:47.797832 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:45515/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39091
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:39:47.911538 12740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:39:47.911803 12740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:39:47.911860 12740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:39:47.915886 12740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:39:47.915943 12740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:39:47.916020 12740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:39:47.921231 12740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:45515/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:39:47.922230 12740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:39:47.923252 12740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:39:47.932997 12746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:39:47.933105 12745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:39:47.933480 12748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:39:47.933640 12740 server_base.cc:1034] running on GCE node
I20260619 14:39:47.934083 12740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:39:47.934823 12740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:39:47.936010 12740 hybrid_clock.cc:648] HybridClock initialized: now 1781879987935982 us; error 41 us; skew 500 ppm
Jun 19 14:39:47 dist-test-slave-32hj krb5kdc[12183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879987, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:39:47.939245 12740 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:39:47.940240 12740 webserver.cc:460] Webserver started at http://127.0.98.1:37833/ using document root <none> and password file <none>
I20260619 14:39:47.940912 12740 fs_manager.cc:359] Metadata directory not provided
I20260619 14:39:47.940992 12740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:39:47.941211 12740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 14:39:47 dist-test-slave-32hj krb5kdc[12183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879987, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 14:39:47.956261 12740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-0/data/instance:
uuid: "ef432f83c6344df6a96f6774b31c2848"
format_stamp: "Formatted at 2026-06-19 14:39:47 on dist-test-slave-32hj"
server_key: "2c6839cff011d42c7e25de8e79a36e49"
server_key_iv: "662b85330783add485c9a2334d562aad"
server_key_version: "kuduclusterkey@0"
I20260619 14:39:47.956871 12740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ef432f83c6344df6a96f6774b31c2848"
format_stamp: "Formatted at 2026-06-19 14:39:47 on dist-test-slave-32hj"
server_key: "2c6839cff011d42c7e25de8e79a36e49"
server_key_iv: "662b85330783add485c9a2334d562aad"
server_key_version: "kuduclusterkey@0"
I20260619 14:39:47.961796 12740 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260619 14:39:47.980129 12757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:39:47.981873 12740 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:39:47.982069 12740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-0/wal
uuid: "ef432f83c6344df6a96f6774b31c2848"
format_stamp: "Formatted at 2026-06-19 14:39:47 on dist-test-slave-32hj"
server_key: "2c6839cff011d42c7e25de8e79a36e49"
server_key_iv: "662b85330783add485c9a2334d562aad"
server_key_version: "kuduclusterkey@0"
I20260619 14:39:47.982290 12740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:39:48.015594 12740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:39:48.019233 12740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:39:48.019414 12740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:39:48.020006 12740 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:39:48.020848 12740 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:39:48.020893 12740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:39:48.020932 12740 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:39:48.020960 12740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:39:48.031167 12740 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44493
I20260619 14:39:48.031373 12870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44493 every 8 connection(s)
I20260619 14:39:48.032272 12740 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:39:48.035569 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 12740
I20260619 14:39:48.035683 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:39:48.036003 392 external_mini_cluster.cc:1428] Setting key 064213e5da3bfe06540ff4a453894463
Jun 19 14:39:48 dist-test-slave-32hj krb5kdc[12183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879987, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 14:39:48.060218 12873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:48.034904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39091 (local address 127.0.98.1:50439)
0619 14:39:48.039689 (+ 4785us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:48.039754 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0619 14:39:48.041040 (+ 1286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:39:48.043188 (+ 2148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:39:48.043197 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:39:48.043570 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:39:48.044270 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:48.044289 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:48.045611 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:48.045614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:48.046951 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:48.046963 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:48.047595 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:48.047599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:48.047790 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:48.048488 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:39:48.048511 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:39:48.050287 (+ 1776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:39:48.056637 (+ 6350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:39:48.056645 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:39:48.056666 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:39:48.057093 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:39:48.057435 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:39:48.057438 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:39:48.057440 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:39:48.057637 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:39:48.058146 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:39:48.058154 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:39:48.058543 (+ 389us) client_negotiation.cc:769] Sending connection context
0619 14:39:48.058741 (+ 198us) client_negotiation.cc:240] Negotiation successful
0619 14:39:48.059102 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":143,"threads_started":1}
I20260619 14:39:48.061758 12874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:48.035920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50439 (local address 127.0.98.62:39091)
0619 14:39:48.042739 (+ 6819us) server_negotiation.cc:206] Beginning negotiation
0619 14:39:48.042748 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:39:48.042764 (+ 16us) server_negotiation.cc:407] Connection header received
0619 14:39:48.042805 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:39:48.042808 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:39:48.042859 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:39:48.044419 (+ 1560us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:39:48.044542 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:48.045431 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:48.047094 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:48.047425 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:48.048123 (+ 698us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:48.052832 (+ 4709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:39:48.052836 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:39:48.052839 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:39:48.052868 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:39:48.056469 (+ 3601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:48.057232 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:48.057235 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:48.057237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:48.057330 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:48.057759 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:48.057762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:48.057764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:48.058018 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:39:48.061346 (+ 3328us) server_negotiation.cc:1035] Waiting for connection context
0619 14:39:48.061434 (+ 88us) server_negotiation.cc:299] Negotiation successful
0619 14:39:48.061581 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6728,"thread_start_us":99,"threads_started":1}
I20260619 14:39:48.063271 12871 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39091
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 14:39:48.063601 12871 heartbeater.cc:464] Registering TS with master...
I20260619 14:39:48.064193 12871 heartbeater.cc:511] Master 127.0.98.62:39091 requested a full tablet report, sending...
I20260619 14:39:48.066043 12659 ts_manager.cc:194] Registered new tserver with Master: ef432f83c6344df6a96f6774b31c2848 (127.0.98.1:44493)
I20260619 14:39:48.067296 12659 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50439
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 14:39:48.081691 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:45515/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39091
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:39:48 dist-test-slave-32hj krb5kdc[12183](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781879988, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 14:39:48.218411 12879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:39:48.218802 12879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:39:48.218895 12879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:39:48.224874 12879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:39:48.224956 12879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:39:48.225042 12879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:39:48.233002 12879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:45515/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:39:48.234452 12879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:39:48.235805 12879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:39:48.247694 12886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:39:48.247774 12888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:39:48.247884 12885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:39:48.249055 12879 server_base.cc:1034] running on GCE node
I20260619 14:39:48.249521 12879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:39:48.250303 12879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:39:48.251500 12879 hybrid_clock.cc:648] HybridClock initialized: now 1781879988251469 us; error 44 us; skew 500 ppm
Jun 19 14:39:48 dist-test-slave-32hj krb5kdc[12183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879988, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:39:48.255010 12879 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:39:48.256114 12879 webserver.cc:460] Webserver started at http://127.0.98.2:40969/ using document root <none> and password file <none>
I20260619 14:39:48.256910 12879 fs_manager.cc:359] Metadata directory not provided
I20260619 14:39:48.257010 12879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:39:48.257297 12879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 14:39:48 dist-test-slave-32hj krb5kdc[12183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879988, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 14:39:48.272073 12879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-1/data/instance:
uuid: "40ccf2a7f1fc48f5a6913e147e2265bb"
format_stamp: "Formatted at 2026-06-19 14:39:48 on dist-test-slave-32hj"
server_key: "9ba4f1d36a092ec87ad32ce54be1c958"
server_key_iv: "dbe41b0f493b3dd2cdc91cffe47f6e2d"
server_key_version: "kuduclusterkey@0"
I20260619 14:39:48.272872 12879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-1/wal/instance:
uuid: "40ccf2a7f1fc48f5a6913e147e2265bb"
format_stamp: "Formatted at 2026-06-19 14:39:48 on dist-test-slave-32hj"
server_key: "9ba4f1d36a092ec87ad32ce54be1c958"
server_key_iv: "dbe41b0f493b3dd2cdc91cffe47f6e2d"
server_key_version: "kuduclusterkey@0"
I20260619 14:39:48.277858 12879 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260619 14:39:48.292987 12895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:39:48.294649 12879 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 14:39:48.294767 12879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-1/wal
uuid: "40ccf2a7f1fc48f5a6913e147e2265bb"
format_stamp: "Formatted at 2026-06-19 14:39:48 on dist-test-slave-32hj"
server_key: "9ba4f1d36a092ec87ad32ce54be1c958"
server_key_iv: "dbe41b0f493b3dd2cdc91cffe47f6e2d"
server_key_version: "kuduclusterkey@0"
I20260619 14:39:48.294884 12879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:39:48.330305 12879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:39:48.333990 12879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:39:48.334210 12879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:39:48.334931 12879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:39:48.335958 12879 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:39:48.336054 12879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:39:48.336138 12879 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:39:48.336187 12879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:39:48.350246 12879 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44263
I20260619 14:39:48.351475 12879 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:39:48.352540 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 12879
I20260619 14:39:48.352651 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:39:48.353022 392 external_mini_cluster.cc:1428] Setting key b18edbf9402304e250f906cf61cbe372
I20260619 14:39:48.354221 13008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44263 every 8 connection(s)
Jun 19 14:39:48 dist-test-slave-32hj krb5kdc[12183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879988, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:39:48.372679 12874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:48.358129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33035 (local address 127.0.98.62:39091)
0619 14:39:48.358246 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 14:39:48.358249 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:39:48.359410 (+ 1161us) server_negotiation.cc:407] Connection header received
0619 14:39:48.360671 (+ 1261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:39:48.360676 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:39:48.360758 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:39:48.360815 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:39:48.362508 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:48.363298 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:48.365032 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:48.365385 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:48.365479 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:48.368462 (+ 2983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:39:48.368465 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:39:48.368469 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:39:48.368499 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:39:48.370223 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:48.370825 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:48.370828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:48.370830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:48.370920 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:48.371333 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:48.371337 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:48.371339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:48.371575 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:39:48.371652 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 14:39:48.372358 (+ 706us) server_negotiation.cc:299] Negotiation successful
0619 14:39:48.372507 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 14:39:48.374548 13012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:48.358673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39091 (local address 127.0.98.2:33035)
0619 14:39:48.359241 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:48.359300 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 14:39:48.360398 (+ 1098us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:39:48.361245 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:39:48.361256 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:39:48.361639 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:39:48.362308 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:48.362329 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:48.363468 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:48.363471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:48.364898 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:48.364909 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:48.365681 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:48.365685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:48.365882 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:48.366563 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:39:48.366583 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:39:48.368258 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:39:48.370359 (+ 2101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:39:48.370364 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:39:48.370380 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:39:48.370706 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:39:48.371036 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:39:48.371039 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:39:48.371041 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:39:48.371223 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:39:48.371737 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:39:48.371743 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:39:48.372067 (+ 324us) client_negotiation.cc:769] Sending connection context
0619 14:39:48.373144 (+ 1077us) client_negotiation.cc:240] Negotiation successful
0619 14:39:48.373493 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":139,"threads_started":1}
I20260619 14:39:48.376060 13009 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39091
I20260619 14:39:48.376287 13009 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 14:39:48.376921 13009 heartbeater.cc:511] Master 127.0.98.62:39091 requested a full tablet report, sending...
I20260619 14:39:48.378016 12659 ts_manager.cc:194] Registered new tserver with Master: 40ccf2a7f1fc48f5a6913e147e2265bb (127.0.98.2:44263)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 14:39:48.378759 12659 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33035
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 14:39:48.409762 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:45515/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39091
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:39:48.569201 13014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:39:48.569622 13014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:39:48.569729 13014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:39:48.575713 13014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:39:48.575878 13014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:39:48.576014 13014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:39:48.583658 13014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:45515/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:39:48.585206 13014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:39:48.586642 13014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:39:48.599153 13020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:39:48.599674 13019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:39:48.600008 13022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:39:48.601114 13014 server_base.cc:1034] running on GCE node
I20260619 14:39:48.601706 13014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:39:48.602444 13014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:39:48.603590 13014 hybrid_clock.cc:648] HybridClock initialized: now 1781879988603525 us; error 82 us; skew 500 ppm
Jun 19 14:39:48 dist-test-slave-32hj krb5kdc[12183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879988, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:39:48.607278 13014 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:39:48.608695 13014 webserver.cc:460] Webserver started at http://127.0.98.3:40977/ using document root <none> and password file <none>
I20260619 14:39:48.609436 13014 fs_manager.cc:359] Metadata directory not provided
I20260619 14:39:48.609563 13014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:39:48.609820 13014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 14:39:48 dist-test-slave-32hj krb5kdc[12183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879988, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 14:39:48.629228 13014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-2/data/instance:
uuid: "ae9f3fcb2c3b464795681d4b4c4179e7"
format_stamp: "Formatted at 2026-06-19 14:39:48 on dist-test-slave-32hj"
server_key: "b4936449af309094ffe12ea82db805e0"
server_key_iv: "2e27a373630707144abd65206e163a94"
server_key_version: "kuduclusterkey@0"
I20260619 14:39:48.630875 13014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ae9f3fcb2c3b464795681d4b4c4179e7"
format_stamp: "Formatted at 2026-06-19 14:39:48 on dist-test-slave-32hj"
server_key: "b4936449af309094ffe12ea82db805e0"
server_key_iv: "2e27a373630707144abd65206e163a94"
server_key_version: "kuduclusterkey@0"
I20260619 14:39:48.635862 13014 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 14:39:48.661074 13029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:39:48.675696 13014 fs_manager.cc:725] Time spent opening block manager: real 0.021s user 0.003s sys 0.000s
I20260619 14:39:48.675873 13014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-2/wal
uuid: "ae9f3fcb2c3b464795681d4b4c4179e7"
format_stamp: "Formatted at 2026-06-19 14:39:48 on dist-test-slave-32hj"
server_key: "b4936449af309094ffe12ea82db805e0"
server_key_iv: "2e27a373630707144abd65206e163a94"
server_key_version: "kuduclusterkey@0"
I20260619 14:39:48.676012 13014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Jun 19 14:39:48 dist-test-slave-32hj krb5kdc[12183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879988, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 14:39:48.719290 13014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:39:48.722800 13014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:39:48.723031 13014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:39:48.723702 13014 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:39:48.724706 13014 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:39:48.724813 13014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:39:48.724902 13014 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:39:48.724947 13014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:39:48.734637 13014 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44495
I20260619 14:39:48.735749 13014 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:39:48.740999 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 13014
I20260619 14:39:48.741101 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:39:48.741377 392 external_mini_cluster.cc:1428] Setting key 9eb94e63851ababed5cb048207922fca
I20260619 14:39:48.759419 13142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44495 every 8 connection(s)
Jun 19 14:39:48 dist-test-slave-32hj krb5kdc[12183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879988, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:39:48.786983 12874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:48.761470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52165 (local address 127.0.98.62:39091)
0619 14:39:48.761591 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 14:39:48.761596 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:39:48.773213 (+ 11617us) server_negotiation.cc:407] Connection header received
0619 14:39:48.774430 (+ 1217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:39:48.774434 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:39:48.774482 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:39:48.774526 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:39:48.776164 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:48.777020 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:48.778643 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:48.778979 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:48.779043 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:48.783205 (+ 4162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:39:48.783209 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:39:48.783211 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:39:48.783235 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:39:48.784802 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:48.785340 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:48.785343 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:48.785345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:48.785431 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:48.785812 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:48.785815 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:48.785817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:48.786011 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:39:48.786075 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 14:39:48.786704 (+ 629us) server_negotiation.cc:299] Negotiation successful
0619 14:39:48.786825 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 14:39:48.788677 13145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:48.760397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39091 (local address 127.0.98.3:52165)
0619 14:39:48.773051 (+ 12654us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:48.773119 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0619 14:39:48.774189 (+ 1070us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:39:48.774988 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:39:48.774998 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:39:48.775354 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:39:48.776016 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:48.776033 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:48.777163 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:48.777166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:48.778521 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:48.778531 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:48.779225 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:48.779229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:48.779411 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:48.781159 (+ 1748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:39:48.781179 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:39:48.783038 (+ 1859us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:39:48.784915 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:39:48.784920 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:39:48.784936 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:39:48.785236 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:39:48.785551 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:39:48.785554 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:39:48.785556 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:39:48.785708 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:39:48.786146 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:39:48.786152 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:39:48.786453 (+ 301us) client_negotiation.cc:769] Sending connection context
0619 14:39:48.787388 (+ 935us) client_negotiation.cc:240] Negotiation successful
0619 14:39:48.787693 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12394,"thread_start_us":147,"threads_started":1}
I20260619 14:39:48.794130 13143 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39091
I20260619 14:39:48.794389 13143 heartbeater.cc:464] Registering TS with master...
I20260619 14:39:48.794966 13143 heartbeater.cc:511] Master 127.0.98.62:39091 requested a full tablet report, sending...
I20260619 14:39:48.796110 12659 ts_manager.cc:194] Registered new tserver with Master: ae9f3fcb2c3b464795681d4b4c4179e7 (127.0.98.3:44495)
I20260619 14:39:48.796867 12659 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52165
I20260619 14:39:48.801407 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:39:48.802160 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 12740
I20260619 14:39:48.810267 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 12879
I20260619 14:39:48.820607 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 13014
I20260619 14:39:48.827064 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 12627
I20260619 14:39:48.832954 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39091
--webserver_interface=127.0.98.62
--webserver_port=39007
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39091
--ranger_config_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:45515/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:45515/kms with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:39:48.939190 13146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:39:48.939416 13146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:39:48.939452 13146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:39:48.943414 13146 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:39:48.943463 13146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:39:48.943481 13146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:39:48.943496 13146 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:39:48.943513 13146 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:39:48.948662 13146 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:45515/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39091
--ranger_config_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=39007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:39:48.949558 13146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:39:48.950469 13146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:39:48.958007 13154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:39:48.958016 13151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:39:48.958087 13146 server_base.cc:1034] running on GCE node
W20260619 14:39:48.958014 13152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:39:48.958802 13146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:39:48.959764 13146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:39:48.960912 13146 hybrid_clock.cc:648] HybridClock initialized: now 1781879988960889 us; error 43 us; skew 500 ppm
Jun 19 14:39:48 dist-test-slave-32hj krb5kdc[12183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879988, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:39:48.963593 13146 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:39:48.964356 13146 webserver.cc:460] Webserver started at http://127.0.98.62:39007/ using document root <none> and password file <none>
I20260619 14:39:48.964891 13146 fs_manager.cc:359] Metadata directory not provided
I20260619 14:39:48.964965 13146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 14:39:49 dist-test-slave-32hj krb5kdc[12183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879988, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 14:39:49.019559 13146 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.002s
I20260619 14:39:49.021780 13162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:39:49.023113 13146 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:39:49.023231 13146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/master-0/wal
uuid: "08cd4a52f1ad4e4cb0c75d4c328ff80d"
format_stamp: "Formatted at 2026-06-19 14:39:47 on dist-test-slave-32hj"
server_key: "ea4940c1d03f42092d95b24f8af0a361"
server_key_iv: "2301a9363fdd5b25c967505c528be288"
server_key_version: "kuduclusterkey@0"
I20260619 14:39:49.023763 13146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:39:49.042708 13146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:39:49.045485 13146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:39:49.045652 13146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:39:49.052492 13146 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39091
I20260619 14:39:49.052510 13214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39091 every 8 connection(s)
I20260619 14:39:49.053551 13146 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:39:49.057617 13215 sys_catalog.cc:263] Verifying existing consensus state
I20260619 14:39:49.059168 13215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08cd4a52f1ad4e4cb0c75d4c328ff80d: Bootstrap starting.
I20260619 14:39:49.059924 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 13146
I20260619 14:39:49.060313 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:45515/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44493
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:39091
--webserver_port=37833
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:45515/kms with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:39:49.074952 13215 log.cc:824] T 00000000000000000000000000000000 P 08cd4a52f1ad4e4cb0c75d4c328ff80d: Log is configured to *not* fsync() on all Append() calls
I20260619 14:39:49.081375 13215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08cd4a52f1ad4e4cb0c75d4c328ff80d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:39:49.081725 13215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08cd4a52f1ad4e4cb0c75d4c328ff80d: Bootstrap complete.
I20260619 14:39:49.085587 13215 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08cd4a52f1ad4e4cb0c75d4c328ff80d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08cd4a52f1ad4e4cb0c75d4c328ff80d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39091 } }
I20260619 14:39:49.086014 13215 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08cd4a52f1ad4e4cb0c75d4c328ff80d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08cd4a52f1ad4e4cb0c75d4c328ff80d, State: Initialized, Role: FOLLOWER
I20260619 14:39:49.086462 13215 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08cd4a52f1ad4e4cb0c75d4c328ff80d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08cd4a52f1ad4e4cb0c75d4c328ff80d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39091 } }
I20260619 14:39:49.086580 13215 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08cd4a52f1ad4e4cb0c75d4c328ff80d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:39:49.086645 13215 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08cd4a52f1ad4e4cb0c75d4c328ff80d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:39:49.086735 13215 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08cd4a52f1ad4e4cb0c75d4c328ff80d [term 1 FOLLOWER]: Advancing to term 2
I20260619 14:39:49.088577 13215 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08cd4a52f1ad4e4cb0c75d4c328ff80d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08cd4a52f1ad4e4cb0c75d4c328ff80d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39091 } }
I20260619 14:39:49.088919 13215 leader_election.cc:302] T 00000000000000000000000000000000 P 08cd4a52f1ad4e4cb0c75d4c328ff80d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08cd4a52f1ad4e4cb0c75d4c328ff80d; no voters:
I20260619 14:39:49.089215 13215 leader_election.cc:288] T 00000000000000000000000000000000 P 08cd4a52f1ad4e4cb0c75d4c328ff80d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 14:39:49.089310 13219 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08cd4a52f1ad4e4cb0c75d4c328ff80d [term 2 FOLLOWER]: Leader election won for term 2
I20260619 14:39:49.089545 13219 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08cd4a52f1ad4e4cb0c75d4c328ff80d [term 2 LEADER]: Becoming Leader. State: Replica: 08cd4a52f1ad4e4cb0c75d4c328ff80d, State: Running, Role: LEADER
I20260619 14:39:49.089895 13219 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08cd4a52f1ad4e4cb0c75d4c328ff80d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08cd4a52f1ad4e4cb0c75d4c328ff80d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39091 } }
I20260619 14:39:49.090466 13215 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08cd4a52f1ad4e4cb0c75d4c328ff80d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:39:49.091215 13220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08cd4a52f1ad4e4cb0c75d4c328ff80d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "08cd4a52f1ad4e4cb0c75d4c328ff80d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08cd4a52f1ad4e4cb0c75d4c328ff80d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39091 } } }
I20260619 14:39:49.091334 13220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08cd4a52f1ad4e4cb0c75d4c328ff80d [sys.catalog]: This master's current role is: LEADER
I20260619 14:39:49.091694 13221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08cd4a52f1ad4e4cb0c75d4c328ff80d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08cd4a52f1ad4e4cb0c75d4c328ff80d. Latest consensus state: current_term: 2 leader_uuid: "08cd4a52f1ad4e4cb0c75d4c328ff80d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08cd4a52f1ad4e4cb0c75d4c328ff80d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39091 } } }
I20260619 14:39:49.091836 13221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08cd4a52f1ad4e4cb0c75d4c328ff80d [sys.catalog]: This master's current role is: LEADER
I20260619 14:39:49.092248 13228 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:39:49.092641 13215 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 14:39:49.098296 13228 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:39:49.098987 13228 catalog_manager.cc:1259] Loaded cluster ID: fbdb41c28eda432e95af34becf4a1294
I20260619 14:39:49.099035 13228 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:39:49.101662 13228 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:39:49.102409 13228 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 08cd4a52f1ad4e4cb0c75d4c328ff80d: Loaded TSK: 0
I20260619 14:39:49.103652 13228 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 14:39:49.197952 13217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:39:49.198366 13217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:39:49.198480 13217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:39:49.204349 13217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:39:49.204428 13217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:39:49.204512 13217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:39:49.212080 13217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:45515/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44493
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37833
--tserver_master_addrs=127.0.98.62:39091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:39:49.213490 13217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:39:49.214792 13217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:39:49.227037 13252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:39:49.227037 13255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:39:49.228044 13253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:39:49.228195 13217 server_base.cc:1034] running on GCE node
I20260619 14:39:49.228674 13217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:39:49.229290 13217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:39:49.236657 13217 hybrid_clock.cc:648] HybridClock initialized: now 1781879989236620 us; error 37 us; skew 500 ppm
Jun 19 14:39:49 dist-test-slave-32hj krb5kdc[12183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:39:49.239439 13217 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:39:49.240432 13217 webserver.cc:460] Webserver started at http://127.0.98.1:37833/ using document root <none> and password file <none>
I20260619 14:39:49.241017 13217 fs_manager.cc:359] Metadata directory not provided
I20260619 14:39:49.241106 13217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 14:39:49 dist-test-slave-32hj krb5kdc[12183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 14:39:49.269145 13217 fs_manager.cc:709] Time spent opening directory manager: real 0.010s user 0.003s sys 0.000s
I20260619 14:39:49.272226 13263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:39:49.273782 13217 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:39:49.273895 13217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-0/wal
uuid: "ef432f83c6344df6a96f6774b31c2848"
format_stamp: "Formatted at 2026-06-19 14:39:47 on dist-test-slave-32hj"
server_key: "2c6839cff011d42c7e25de8e79a36e49"
server_key_iv: "662b85330783add485c9a2334d562aad"
server_key_version: "kuduclusterkey@0"
I20260619 14:39:49.274582 13217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:39:49.303844 13217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:39:49.306746 13217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:39:49.306921 13217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:39:49.307507 13217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:39:49.308545 13217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:39:49.308640 13217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:39:49.308761 13217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:39:49.308823 13217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:39:49.320426 13217 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44493
I20260619 14:39:49.321800 13217 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:39:49.323385 13376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44493 every 8 connection(s)
I20260619 14:39:49.323693 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 13217
I20260619 14:39:49.324111 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:45515/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:44263
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:39091
--webserver_port=40969
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:45515/kms with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:39:49 dist-test-slave-32hj krb5kdc[12183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:39:49.346509 13380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:49.325066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39091 (local address 127.0.98.1:43281)
0619 14:39:49.328445 (+ 3379us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:49.328503 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 14:39:49.329756 (+ 1253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:39:49.331348 (+ 1592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:39:49.331357 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:39:49.331672 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:39:49.332190 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:49.332207 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:49.333592 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:49.333595 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:49.334818 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:49.334829 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:49.336060 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:49.336064 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:49.336261 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:49.338747 (+ 2486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:39:49.338768 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:39:49.340363 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:39:49.343034 (+ 2671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:39:49.343040 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:39:49.343057 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:39:49.343405 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:39:49.343813 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:39:49.343816 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:39:49.343818 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:39:49.344006 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:39:49.344600 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:39:49.344606 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:39:49.344970 (+ 364us) client_negotiation.cc:769] Sending connection context
0619 14:39:49.345146 (+ 176us) client_negotiation.cc:240] Negotiation successful
0619 14:39:49.345498 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3123,"thread_start_us":112,"threads_started":1}
I20260619 14:39:49.346500 13379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:49.324833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43281 (local address 127.0.98.62:39091)
0619 14:39:49.325417 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0619 14:39:49.325448 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0619 14:39:49.328803 (+ 3355us) server_negotiation.cc:407] Connection header received
0619 14:39:49.330392 (+ 1589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:39:49.330409 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:39:49.330945 (+ 536us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:39:49.331262 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:39:49.332402 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:49.333465 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:49.334959 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:49.335677 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:49.335842 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:49.340563 (+ 4721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:39:49.340570 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:39:49.340583 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:39:49.340610 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:39:49.342890 (+ 2280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:49.343545 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:49.343552 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:49.343558 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:49.343699 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:49.344097 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:49.344100 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:49.344102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:49.344485 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:39:49.344607 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 14:39:49.345279 (+ 672us) server_negotiation.cc:299] Negotiation successful
0619 14:39:49.345519 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":112,"threads_started":1}
I20260619 14:39:49.350152 13377 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39091
I20260619 14:39:49.350399 13377 heartbeater.cc:464] Registering TS with master...
I20260619 14:39:49.350862 13377 heartbeater.cc:511] Master 127.0.98.62:39091 requested a full tablet report, sending...
W20260619 14:39:49.353067 13377 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39091 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 14:39:49.353189 13377 heartbeater.cc:464] Registering TS with master...
I20260619 14:39:49.353369 13377 heartbeater.cc:511] Master 127.0.98.62:39091 requested a full tablet report, sending...
I20260619 14:39:49.354298 13377 heartbeater.cc:464] Registering TS with master...
I20260619 14:39:49.354466 13377 heartbeater.cc:511] Master 127.0.98.62:39091 requested a full tablet report, sending...
W20260619 14:39:49.355448 13377 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 14:39:49.483180 13381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:39:49.483582 13381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:39:49.483681 13381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:39:49.489696 13381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:39:49.489784 13381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:39:49.489890 13381 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:39:49.497680 13381 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:45515/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:44263
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=40969
--tserver_master_addrs=127.0.98.62:39091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13381
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:39:49.499045 13381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:39:49.500422 13381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:39:49.512764 13389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:39:49.512970 13387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:39:49.512758 13386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:39:49.513221 13381 server_base.cc:1034] running on GCE node
I20260619 14:39:49.514062 13381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:39:49.514904 13381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:39:49.518247 13381 hybrid_clock.cc:648] HybridClock initialized: now 1781879989518209 us; error 46 us; skew 500 ppm
Jun 19 14:39:49 dist-test-slave-32hj krb5kdc[12183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:39:49.521762 13381 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:39:49.522972 13381 webserver.cc:460] Webserver started at http://127.0.98.2:40969/ using document root <none> and password file <none>
I20260619 14:39:49.523720 13381 fs_manager.cc:359] Metadata directory not provided
I20260619 14:39:49.523869 13381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 14:39:49 dist-test-slave-32hj krb5kdc[12183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 14:39:49.556782 13381 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 14:39:49.560009 13397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:39:49.561679 13381 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260619 14:39:49.561832 13381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-1/wal
uuid: "40ccf2a7f1fc48f5a6913e147e2265bb"
format_stamp: "Formatted at 2026-06-19 14:39:48 on dist-test-slave-32hj"
server_key: "9ba4f1d36a092ec87ad32ce54be1c958"
server_key_iv: "dbe41b0f493b3dd2cdc91cffe47f6e2d"
server_key_version: "kuduclusterkey@0"
I20260619 14:39:49.562556 13381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:39:49.597847 13381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:39:49.600652 13381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:39:49.600850 13381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:39:49.601454 13381 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:39:49.602435 13381 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:39:49.602507 13381 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:39:49.602558 13381 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:39:49.602586 13381 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:39:49.612843 13381 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44263
I20260619 14:39:49.612870 13511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44263 every 8 connection(s)
I20260619 14:39:49.614017 13381 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:39:49.623843 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 13381
I20260619 14:39:49.624377 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:45515/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:44495
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:39091
--webserver_port=40977
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:45515/kms with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:39:49 dist-test-slave-32hj krb5kdc[12183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:39:49.631166 13514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:49.616482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39091 (local address 127.0.98.2:55311)
0619 14:39:49.617018 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:49.617072 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 14:39:49.618140 (+ 1068us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:39:49.618904 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:39:49.618913 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:39:49.619237 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:39:49.619880 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:49.619899 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:49.622064 (+ 2165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:49.622067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:49.623062 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:49.623072 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:49.623534 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:49.623537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:49.623709 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:49.624263 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:39:49.624283 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:39:49.626027 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:39:49.627981 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:39:49.627987 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:39:49.628001 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:39:49.628309 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:39:49.628589 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:39:49.628592 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:39:49.628594 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:39:49.628781 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:39:49.629262 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:39:49.629269 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:39:49.629588 (+ 319us) client_negotiation.cc:769] Sending connection context
0619 14:39:49.629785 (+ 197us) client_negotiation.cc:240] Negotiation successful
0619 14:39:49.630124 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":163,"threads_started":1}
I20260619 14:39:49.631887 13379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:49.615928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55311 (local address 127.0.98.62:39091)
0619 14:39:49.616053 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 14:39:49.616056 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:39:49.617177 (+ 1121us) server_negotiation.cc:407] Connection header received
0619 14:39:49.618385 (+ 1208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:39:49.618389 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:39:49.618460 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:39:49.618514 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:39:49.620052 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:49.621933 (+ 1881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:49.623173 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:49.623395 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:49.623484 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:49.626189 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:39:49.626194 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:39:49.626197 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:39:49.626230 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:39:49.627849 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:49.628413 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:49.628417 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:49.628419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:49.628501 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:49.628887 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:49.628890 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:49.628891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:49.629117 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:39:49.629207 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 14:39:49.631602 (+ 2395us) server_negotiation.cc:299] Negotiation successful
0619 14:39:49.631728 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 14:39:49.633087 13512 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39091
I20260619 14:39:49.633384 13512 heartbeater.cc:464] Registering TS with master...
I20260619 14:39:49.634287 13512 heartbeater.cc:511] Master 127.0.98.62:39091 requested a full tablet report, sending...
W20260619 14:39:49.636018 13512 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39091 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 14:39:49.636178 13512 heartbeater.cc:464] Registering TS with master...
I20260619 14:39:49.636354 13512 heartbeater.cc:511] Master 127.0.98.62:39091 requested a full tablet report, sending...
I20260619 14:39:49.637166 13512 heartbeater.cc:464] Registering TS with master...
I20260619 14:39:49.637357 13512 heartbeater.cc:511] Master 127.0.98.62:39091 requested a full tablet report, sending...
W20260619 14:39:49.638078 13512 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 14:39:49.768049 13515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:39:49.768474 13515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:39:49.768566 13515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:39:49.774586 13515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:39:49.774682 13515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:39:49.774784 13515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:39:49.782667 13515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:45515/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:44495
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40977
--tserver_master_addrs=127.0.98.62:39091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:39:49.784082 13515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:39:49.785542 13515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:39:49.796887 13521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:39:49.796933 13523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:39:49.796936 13520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:39:49.797252 13515 server_base.cc:1034] running on GCE node
I20260619 14:39:49.797737 13515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:39:49.798367 13515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:39:49.799499 13515 hybrid_clock.cc:648] HybridClock initialized: now 1781879989799469 us; error 47 us; skew 500 ppm
Jun 19 14:39:49 dist-test-slave-32hj krb5kdc[12183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:39:49.802258 13515 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:39:49.803156 13515 webserver.cc:460] Webserver started at http://127.0.98.3:40977/ using document root <none> and password file <none>
I20260619 14:39:49.803736 13515 fs_manager.cc:359] Metadata directory not provided
I20260619 14:39:49.803828 13515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 14:39:49 dist-test-slave-32hj krb5kdc[12183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 14:39:49.822695 13515 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 14:39:49.826582 13531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:39:49.829778 13515 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.000s
I20260619 14:39:49.829908 13515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-2/wal
uuid: "ae9f3fcb2c3b464795681d4b4c4179e7"
format_stamp: "Formatted at 2026-06-19 14:39:48 on dist-test-slave-32hj"
server_key: "b4936449af309094ffe12ea82db805e0"
server_key_iv: "2e27a373630707144abd65206e163a94"
server_key_version: "kuduclusterkey@0"
I20260619 14:39:49.830634 13515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:39:49.854441 13515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:39:49.857617 13515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:39:49.857810 13515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:39:49.858393 13515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:39:49.859416 13515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:39:49.859472 13515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:39:49.859530 13515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:39:49.859563 13515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:39:49.870684 13515 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44495
I20260619 14:39:49.871670 13651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44495 every 8 connection(s)
I20260619 14:39:49.872104 13515 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:39:49.873782 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 13515
Jun 19 14:39:49 dist-test-slave-32hj krb5kdc[12183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:39:49.885669 13656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:49.875676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41756 (local address 127.0.98.62:39091)
0619 14:39:49.875922 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0619 14:39:49.875931 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:39:49.875944 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:39:49.875992 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:39:49.875995 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:39:49.876045 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:39:49.876145 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:39:49.877750 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:49.878541 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:49.879905 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:49.880357 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:49.880461 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:49.881945 (+ 1484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:39:49.881949 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:39:49.881952 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:39:49.881976 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:39:49.883352 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:49.883989 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:49.883993 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:49.883995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:49.884112 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:49.884809 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:49.884811 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:49.884813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:49.885004 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:39:49.885189 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0619 14:39:49.885379 (+ 190us) server_negotiation.cc:299] Negotiation successful
0619 14:39:49.885495 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":188,"thread_start_us":130,"threads_started":1}
I20260619 14:39:49.887487 13379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:49.875593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36095 (local address 127.0.98.62:39091)
0619 14:39:49.875712 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 14:39:49.875718 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:39:49.875737 (+ 19us) server_negotiation.cc:407] Connection header received
0619 14:39:49.876822 (+ 1085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:39:49.876826 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:39:49.876867 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:39:49.876947 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:39:49.878145 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:49.878783 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:49.879925 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:49.880450 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:49.880552 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:49.883468 (+ 2916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:39:49.883472 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:39:49.883475 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:39:49.883497 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:39:49.885012 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:49.885707 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:49.885711 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:49.885712 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:49.885795 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:49.886364 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:49.886367 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:49.886369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:49.886565 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:39:49.886630 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 14:39:49.887283 (+ 653us) server_negotiation.cc:299] Negotiation successful
0619 14:39:49.887374 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":84,"server-negotiator.queue_time_us":48}
I20260619 14:39:49.888536 13655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:49.874576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39091 (local address 127.0.98.3:36095)
0619 14:39:49.875527 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:49.875582 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 14:39:49.876562 (+ 980us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:39:49.877159 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:39:49.877168 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:39:49.877429 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:39:49.877984 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:49.878004 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:49.878897 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:49.878899 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:49.879809 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:49.879819 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:49.880714 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:49.880752 (+ 38us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:49.880959 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:49.881486 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:39:49.881505 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:39:49.883304 (+ 1799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:39:49.885136 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:39:49.885142 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:39:49.885158 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:39:49.885472 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:39:49.885910 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:39:49.885913 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:39:49.885927 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:39:49.886253 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:39:49.886706 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:39:49.886712 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:39:49.887036 (+ 324us) client_negotiation.cc:769] Sending connection context
0619 14:39:49.887246 (+ 210us) client_negotiation.cc:240] Negotiation successful
0619 14:39:49.887561 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":733,"thread_start_us":128,"threads_started":1}
I20260619 14:39:49.889602 13652 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39091
I20260619 14:39:49.889861 13652 heartbeater.cc:464] Registering TS with master...
I20260619 14:39:49.890489 13652 heartbeater.cc:511] Master 127.0.98.62:39091 requested a full tablet report, sending...
I20260619 14:39:49.891880 13179 ts_manager.cc:194] Registered new tserver with Master: ae9f3fcb2c3b464795681d4b4c4179e7 (127.0.98.3:44495)
I20260619 14:39:49.893136 13179 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36095
Jun 19 14:39:50 dist-test-slave-32hj krb5kdc[12183](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781879990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:39:50.355671 13377 heartbeater.cc:464] Registering TS with master...
I20260619 14:39:50.355979 13377 heartbeater.cc:511] Master 127.0.98.62:39091 requested a full tablet report, sending...
I20260619 14:39:50.356833 13179 ts_manager.cc:194] Registered new tserver with Master: ef432f83c6344df6a96f6774b31c2848 (127.0.98.1:44493)
I20260619 14:39:50.357589 13179 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43281
Jun 19 14:39:50 dist-test-slave-32hj krb5kdc[12183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 14:39:50.638321 13512 heartbeater.cc:464] Registering TS with master...
I20260619 14:39:50.638612 13512 heartbeater.cc:511] Master 127.0.98.62:39091 requested a full tablet report, sending...
I20260619 14:39:50.639375 13179 ts_manager.cc:194] Registered new tserver with Master: 40ccf2a7f1fc48f5a6913e147e2265bb (127.0.98.2:44263)
I20260619 14:39:50.640137 13179 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55311
I20260619 14:39:50.643175 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:39:50.653784 13667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:50.644781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41762 (local address 127.0.98.62:39091)
0619 14:39:50.645310 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0619 14:39:50.645315 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:39:50.645327 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:39:50.645360 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:39:50.645363 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:39:50.645406 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:39:50.645484 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:39:50.646313 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:50.647118 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:50.648236 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:50.648588 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:50.648682 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:50.650316 (+ 1634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:39:50.650320 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:39:50.650323 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:39:50.650348 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:39:50.652032 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:50.652479 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:50.652482 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:50.652484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:50.652565 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:50.652835 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:50.652837 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:50.652839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:50.653056 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:39:50.653171 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 14:39:50.653511 (+ 340us) server_negotiation.cc:299] Negotiation successful
0619 14:39:50.653624 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":75,"threads_started":1}
I20260619 14:39:50.657477 13179 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41762:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:39:50.662017 13179 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:39:50.680552 13672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:50.672773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44263 (local address 127.0.0.1:54214)
0619 14:39:50.674780 (+ 2007us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:50.674824 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 14:39:50.674959 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:39:50.675805 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:39:50.675811 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:39:50.675834 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:39:50.675959 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:50.675965 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:50.677383 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:50.677386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:50.678757 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:50.678768 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:50.679919 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:50.679924 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:50.680033 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:50.680166 (+ 133us) client_negotiation.cc:769] Sending connection context
0619 14:39:50.680279 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 14:39:50.680402 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1848,"thread_start_us":93,"threads_started":1}
I20260619 14:39:50.680554 13673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:50.673445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54214 (local address 127.0.98.2:44263)
0619 14:39:50.673751 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0619 14:39:50.673758 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:39:50.674923 (+ 1165us) server_negotiation.cc:407] Connection header received
0619 14:39:50.675218 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:39:50.675224 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:39:50.675431 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:39:50.675574 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:39:50.676075 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:50.677264 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:50.678992 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:50.679802 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:50.679916 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:50.680053 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 14:39:50.680312 (+ 259us) server_negotiation.cc:299] Negotiation successful
0619 14:39:50.680410 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":98,"threads_started":1}
I20260619 14:39:50.683974 13446 tablet_service.cc:1467] Processing CreateTablet for tablet 301cab3b867a440591bc6a7e67c380fb (DEFAULT_TABLE table=test-table [id=e367b1086cfc4c64930ce38558ad5fa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:39:50.684996 13446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 301cab3b867a440591bc6a7e67c380fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:39:50.687484 13672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:50.674169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44493 (local address 127.0.0.1:34584)
0619 14:39:50.681018 (+ 6849us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:50.681031 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:39:50.681104 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:39:50.681786 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:39:50.681789 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:39:50.681806 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:39:50.681887 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:50.681894 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:50.684301 (+ 2407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:50.684305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:50.685408 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:50.685417 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:50.687137 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:50.687140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:50.687223 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:50.687244 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 14:39:50.687302 (+ 58us) client_negotiation.cc:240] Negotiation successful
0619 14:39:50.687363 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6794,"thread_start_us":80,"threads_started":1}
I20260619 14:39:50.688201 13676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:50.674515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34584 (local address 127.0.98.1:44493)
0619 14:39:50.674749 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 14:39:50.674756 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:39:50.681191 (+ 6435us) server_negotiation.cc:407] Connection header received
0619 14:39:50.681270 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:39:50.681275 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:39:50.681476 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:39:50.681623 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:39:50.682633 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:50.684183 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:50.685526 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:50.686276 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:50.686373 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:50.687937 (+ 1564us) server_negotiation.cc:1035] Waiting for connection context
0619 14:39:50.687984 (+ 47us) server_negotiation.cc:299] Negotiation successful
0619 14:39:50.688071 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":71,"threads_started":1}
I20260619 14:39:50.691072 13311 tablet_service.cc:1467] Processing CreateTablet for tablet 301cab3b867a440591bc6a7e67c380fb (DEFAULT_TABLE table=test-table [id=e367b1086cfc4c64930ce38558ad5fa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:39:50.692106 13311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 301cab3b867a440591bc6a7e67c380fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:39:50.693255 13678 tablet_bootstrap.cc:492] T 301cab3b867a440591bc6a7e67c380fb P 40ccf2a7f1fc48f5a6913e147e2265bb: Bootstrap starting.
I20260619 14:39:50.695453 13678 tablet_bootstrap.cc:654] T 301cab3b867a440591bc6a7e67c380fb P 40ccf2a7f1fc48f5a6913e147e2265bb: Neither blocks nor log segments found. Creating new log.
I20260619 14:39:50.696288 13678 log.cc:824] T 301cab3b867a440591bc6a7e67c380fb P 40ccf2a7f1fc48f5a6913e147e2265bb: Log is configured to *not* fsync() on all Append() calls
I20260619 14:39:50.699664 13674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:50.674361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44495 (local address 127.0.0.1:51854)
0619 14:39:50.684854 (+ 10493us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:50.684867 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:39:50.684946 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:39:50.686102 (+ 1156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:39:50.686104 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:39:50.686121 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:39:50.686207 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:50.686213 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:50.696795 (+ 10582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:50.696799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:50.697987 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:50.697998 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:50.699258 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:50.699261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:50.699364 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:50.699391 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 14:39:50.699456 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 14:39:50.699527 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10432,"thread_start_us":83,"threads_started":1}
I20260619 14:39:50.700176 13677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:50.674523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51854 (local address 127.0.98.3:44495)
0619 14:39:50.685698 (+ 11175us) server_negotiation.cc:206] Beginning negotiation
0619 14:39:50.685705 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:39:50.685727 (+ 22us) server_negotiation.cc:407] Connection header received
0619 14:39:50.685795 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:39:50.685799 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:39:50.685984 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:39:50.686116 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:39:50.692838 (+ 6722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:50.693983 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:50.698140 (+ 4157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:50.698964 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:50.699063 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:50.699200 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 14:39:50.699937 (+ 737us) server_negotiation.cc:299] Negotiation successful
0619 14:39:50.700039 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11070,"thread_start_us":72,"threads_started":1}
I20260619 14:39:50.700314 13680 tablet_bootstrap.cc:492] T 301cab3b867a440591bc6a7e67c380fb P ef432f83c6344df6a96f6774b31c2848: Bootstrap starting.
I20260619 14:39:50.702488 13680 tablet_bootstrap.cc:654] T 301cab3b867a440591bc6a7e67c380fb P ef432f83c6344df6a96f6774b31c2848: Neither blocks nor log segments found. Creating new log.
I20260619 14:39:50.703289 13680 log.cc:824] T 301cab3b867a440591bc6a7e67c380fb P ef432f83c6344df6a96f6774b31c2848: Log is configured to *not* fsync() on all Append() calls
I20260619 14:39:50.702968 13560 tablet_service.cc:1467] Processing CreateTablet for tablet 301cab3b867a440591bc6a7e67c380fb (DEFAULT_TABLE table=test-table [id=e367b1086cfc4c64930ce38558ad5fa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:39:50.704069 13560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 301cab3b867a440591bc6a7e67c380fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:39:50.705307 13680 tablet_bootstrap.cc:492] T 301cab3b867a440591bc6a7e67c380fb P ef432f83c6344df6a96f6774b31c2848: No bootstrap required, opened a new log
I20260619 14:39:50.705505 13680 ts_tablet_manager.cc:1397] T 301cab3b867a440591bc6a7e67c380fb P ef432f83c6344df6a96f6774b31c2848: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:39:50.710163 13680 raft_consensus.cc:348] T 301cab3b867a440591bc6a7e67c380fb P ef432f83c6344df6a96f6774b31c2848 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40ccf2a7f1fc48f5a6913e147e2265bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44263 } } peers { permanent_uuid: "ef432f83c6344df6a96f6774b31c2848" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44493 } } peers { permanent_uuid: "ae9f3fcb2c3b464795681d4b4c4179e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44495 } }
I20260619 14:39:50.710434 13680 raft_consensus.cc:374] T 301cab3b867a440591bc6a7e67c380fb P ef432f83c6344df6a96f6774b31c2848 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:39:50.710494 13680 raft_consensus.cc:729] T 301cab3b867a440591bc6a7e67c380fb P ef432f83c6344df6a96f6774b31c2848 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef432f83c6344df6a96f6774b31c2848, State: Initialized, Role: FOLLOWER
I20260619 14:39:50.711030 13680 consensus_queue.cc:260] T 301cab3b867a440591bc6a7e67c380fb P ef432f83c6344df6a96f6774b31c2848 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40ccf2a7f1fc48f5a6913e147e2265bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44263 } } peers { permanent_uuid: "ef432f83c6344df6a96f6774b31c2848" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44493 } } peers { permanent_uuid: "ae9f3fcb2c3b464795681d4b4c4179e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44495 } }
I20260619 14:39:50.712209 13680 ts_tablet_manager.cc:1428] T 301cab3b867a440591bc6a7e67c380fb P ef432f83c6344df6a96f6774b31c2848: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260619 14:39:50.712555 13377 heartbeater.cc:503] Master 127.0.98.62:39091 was elected leader, sending a full tablet report...
I20260619 14:39:50.714453 13678 tablet_bootstrap.cc:492] T 301cab3b867a440591bc6a7e67c380fb P 40ccf2a7f1fc48f5a6913e147e2265bb: No bootstrap required, opened a new log
I20260619 14:39:50.714671 13678 ts_tablet_manager.cc:1397] T 301cab3b867a440591bc6a7e67c380fb P 40ccf2a7f1fc48f5a6913e147e2265bb: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20260619 14:39:50.716117 13683 tablet_bootstrap.cc:492] T 301cab3b867a440591bc6a7e67c380fb P ae9f3fcb2c3b464795681d4b4c4179e7: Bootstrap starting.
I20260619 14:39:50.718771 13678 raft_consensus.cc:348] T 301cab3b867a440591bc6a7e67c380fb P 40ccf2a7f1fc48f5a6913e147e2265bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40ccf2a7f1fc48f5a6913e147e2265bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44263 } } peers { permanent_uuid: "ef432f83c6344df6a96f6774b31c2848" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44493 } } peers { permanent_uuid: "ae9f3fcb2c3b464795681d4b4c4179e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44495 } }
I20260619 14:39:50.719039 13678 raft_consensus.cc:374] T 301cab3b867a440591bc6a7e67c380fb P 40ccf2a7f1fc48f5a6913e147e2265bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:39:50.719096 13678 raft_consensus.cc:729] T 301cab3b867a440591bc6a7e67c380fb P 40ccf2a7f1fc48f5a6913e147e2265bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40ccf2a7f1fc48f5a6913e147e2265bb, State: Initialized, Role: FOLLOWER
I20260619 14:39:50.719594 13678 consensus_queue.cc:260] T 301cab3b867a440591bc6a7e67c380fb P 40ccf2a7f1fc48f5a6913e147e2265bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40ccf2a7f1fc48f5a6913e147e2265bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44263 } } peers { permanent_uuid: "ef432f83c6344df6a96f6774b31c2848" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44493 } } peers { permanent_uuid: "ae9f3fcb2c3b464795681d4b4c4179e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44495 } }
I20260619 14:39:50.720604 13678 ts_tablet_manager.cc:1428] T 301cab3b867a440591bc6a7e67c380fb P 40ccf2a7f1fc48f5a6913e147e2265bb: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 14:39:50.720820 13683 tablet_bootstrap.cc:654] T 301cab3b867a440591bc6a7e67c380fb P ae9f3fcb2c3b464795681d4b4c4179e7: Neither blocks nor log segments found. Creating new log.
I20260619 14:39:50.720956 13512 heartbeater.cc:503] Master 127.0.98.62:39091 was elected leader, sending a full tablet report...
I20260619 14:39:50.721623 13683 log.cc:824] T 301cab3b867a440591bc6a7e67c380fb P ae9f3fcb2c3b464795681d4b4c4179e7: Log is configured to *not* fsync() on all Append() calls
I20260619 14:39:50.723740 13683 tablet_bootstrap.cc:492] T 301cab3b867a440591bc6a7e67c380fb P ae9f3fcb2c3b464795681d4b4c4179e7: No bootstrap required, opened a new log
I20260619 14:39:50.723933 13683 ts_tablet_manager.cc:1397] T 301cab3b867a440591bc6a7e67c380fb P ae9f3fcb2c3b464795681d4b4c4179e7: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260619 14:39:50.728181 13683 raft_consensus.cc:348] T 301cab3b867a440591bc6a7e67c380fb P ae9f3fcb2c3b464795681d4b4c4179e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40ccf2a7f1fc48f5a6913e147e2265bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44263 } } peers { permanent_uuid: "ef432f83c6344df6a96f6774b31c2848" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44493 } } peers { permanent_uuid: "ae9f3fcb2c3b464795681d4b4c4179e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44495 } }
I20260619 14:39:50.728461 13683 raft_consensus.cc:374] T 301cab3b867a440591bc6a7e67c380fb P ae9f3fcb2c3b464795681d4b4c4179e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:39:50.728510 13683 raft_consensus.cc:729] T 301cab3b867a440591bc6a7e67c380fb P ae9f3fcb2c3b464795681d4b4c4179e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9f3fcb2c3b464795681d4b4c4179e7, State: Initialized, Role: FOLLOWER
I20260619 14:39:50.728972 13683 consensus_queue.cc:260] T 301cab3b867a440591bc6a7e67c380fb P ae9f3fcb2c3b464795681d4b4c4179e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40ccf2a7f1fc48f5a6913e147e2265bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44263 } } peers { permanent_uuid: "ef432f83c6344df6a96f6774b31c2848" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44493 } } peers { permanent_uuid: "ae9f3fcb2c3b464795681d4b4c4179e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44495 } }
I20260619 14:39:50.729637 13652 heartbeater.cc:503] Master 127.0.98.62:39091 was elected leader, sending a full tablet report...
I20260619 14:39:50.731645 13683 ts_tablet_manager.cc:1428] T 301cab3b867a440591bc6a7e67c380fb P ae9f3fcb2c3b464795681d4b4c4179e7: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260619 14:39:50.746132 13682 raft_consensus.cc:482] T 301cab3b867a440591bc6a7e67c380fb P ef432f83c6344df6a96f6774b31c2848 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:39:50.746364 13682 raft_consensus.cc:504] T 301cab3b867a440591bc6a7e67c380fb P ef432f83c6344df6a96f6774b31c2848 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40ccf2a7f1fc48f5a6913e147e2265bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44263 } } peers { permanent_uuid: "ef432f83c6344df6a96f6774b31c2848" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44493 } } peers { permanent_uuid: "ae9f3fcb2c3b464795681d4b4c4179e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44495 } }
I20260619 14:39:50.747627 13682 leader_election.cc:288] T 301cab3b867a440591bc6a7e67c380fb P ef432f83c6344df6a96f6774b31c2848 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40ccf2a7f1fc48f5a6913e147e2265bb (127.0.98.2:44263), ae9f3fcb2c3b464795681d4b4c4179e7 (127.0.98.3:44495)
I20260619 14:39:50.753801 13677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:50.748324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36033 (local address 127.0.98.3:44495)
0619 14:39:50.748454 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 14:39:50.748459 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:39:50.749364 (+ 905us) server_negotiation.cc:407] Connection header received
0619 14:39:50.749571 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:39:50.749574 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:39:50.749623 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:39:50.749669 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:39:50.750102 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:50.750868 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:50.752462 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:50.753093 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:50.753170 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:50.753230 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 14:39:50.753600 (+ 370us) server_negotiation.cc:299] Negotiation successful
0619 14:39:50.753677 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 14:39:50.754385 13687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:50.747948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44495 (local address 127.0.98.1:36033)
0619 14:39:50.749269 (+ 1321us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:50.749283 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:39:50.749444 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:39:50.749808 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:39:50.749811 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:39:50.749829 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:39:50.749981 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:50.749988 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:50.751007 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:50.751010 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:50.752307 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:50.752318 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:50.753326 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:50.753329 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:50.753454 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:50.753480 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 14:39:50.754146 (+ 666us) client_negotiation.cc:240] Negotiation successful
0619 14:39:50.754221 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1259,"thread_start_us":116,"threads_started":1}
I20260619 14:39:50.755112 13599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "301cab3b867a440591bc6a7e67c380fb" candidate_uuid: "ef432f83c6344df6a96f6774b31c2848" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9f3fcb2c3b464795681d4b4c4179e7" is_pre_election: true
I20260619 14:39:50.755414 13599 raft_consensus.cc:2454] T 301cab3b867a440591bc6a7e67c380fb P ae9f3fcb2c3b464795681d4b4c4179e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef432f83c6344df6a96f6774b31c2848 in term 0.
I20260619 14:39:50.755852 13266 leader_election.cc:302] T 301cab3b867a440591bc6a7e67c380fb P ef432f83c6344df6a96f6774b31c2848 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9f3fcb2c3b464795681d4b4c4179e7, ef432f83c6344df6a96f6774b31c2848; no voters:
I20260619 14:39:50.756135 13682 raft_consensus.cc:2781] T 301cab3b867a440591bc6a7e67c380fb P ef432f83c6344df6a96f6774b31c2848 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:39:50.756208 13682 raft_consensus.cc:482] T 301cab3b867a440591bc6a7e67c380fb P ef432f83c6344df6a96f6774b31c2848 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:39:50.756247 13682 raft_consensus.cc:3037] T 301cab3b867a440591bc6a7e67c380fb P ef432f83c6344df6a96f6774b31c2848 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:39:50.757503 13682 raft_consensus.cc:504] T 301cab3b867a440591bc6a7e67c380fb P ef432f83c6344df6a96f6774b31c2848 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40ccf2a7f1fc48f5a6913e147e2265bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44263 } } peers { permanent_uuid: "ef432f83c6344df6a96f6774b31c2848" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44493 } } peers { permanent_uuid: "ae9f3fcb2c3b464795681d4b4c4179e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44495 } }
I20260619 14:39:50.757735 13673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:50.749094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52495 (local address 127.0.98.2:44263)
0619 14:39:50.749196 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0619 14:39:50.749200 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:39:50.752862 (+ 3662us) server_negotiation.cc:407] Connection header received
0619 14:39:50.753039 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:39:50.753042 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:39:50.753088 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:39:50.753130 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:39:50.753502 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:50.754220 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:50.756428 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:50.757052 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:50.757125 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:50.757184 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 14:39:50.757541 (+ 357us) server_negotiation.cc:299] Negotiation successful
0619 14:39:50.757622 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 14:39:50.757830 13682 leader_election.cc:288] T 301cab3b867a440591bc6a7e67c380fb P ef432f83c6344df6a96f6774b31c2848 [CANDIDATE]: Term 1 election: Requested vote from peers 40ccf2a7f1fc48f5a6913e147e2265bb (127.0.98.2:44263), ae9f3fcb2c3b464795681d4b4c4179e7 (127.0.98.3:44495)
I20260619 14:39:50.758220 13599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "301cab3b867a440591bc6a7e67c380fb" candidate_uuid: "ef432f83c6344df6a96f6774b31c2848" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae9f3fcb2c3b464795681d4b4c4179e7"
I20260619 14:39:50.758292 13688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:50.748759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44263 (local address 127.0.98.1:52495)
0619 14:39:50.752776 (+ 4017us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:50.752790 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:39:50.752928 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:39:50.753251 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:39:50.753254 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:39:50.753270 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:39:50.753380 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:50.753386 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:50.754356 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:50.754360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:50.756171 (+ 1811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:50.756182 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:50.757276 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:50.757279 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:50.757392 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:50.757415 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 14:39:50.758092 (+ 677us) client_negotiation.cc:240] Negotiation successful
0619 14:39:50.758162 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3932,"thread_start_us":87,"threads_started":1}
I20260619 14:39:50.758337 13599 raft_consensus.cc:3037] T 301cab3b867a440591bc6a7e67c380fb P ae9f3fcb2c3b464795681d4b4c4179e7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:39:50.759296 13463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "301cab3b867a440591bc6a7e67c380fb" candidate_uuid: "ef432f83c6344df6a96f6774b31c2848" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40ccf2a7f1fc48f5a6913e147e2265bb"
I20260619 14:39:50.759480 13599 raft_consensus.cc:2454] T 301cab3b867a440591bc6a7e67c380fb P ae9f3fcb2c3b464795681d4b4c4179e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef432f83c6344df6a96f6774b31c2848 in term 1.
I20260619 14:39:50.759524 13463 raft_consensus.cc:3037] T 301cab3b867a440591bc6a7e67c380fb P 40ccf2a7f1fc48f5a6913e147e2265bb [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:39:50.759744 13266 leader_election.cc:302] T 301cab3b867a440591bc6a7e67c380fb P ef432f83c6344df6a96f6774b31c2848 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9f3fcb2c3b464795681d4b4c4179e7, ef432f83c6344df6a96f6774b31c2848; no voters:
I20260619 14:39:50.759884 13682 raft_consensus.cc:2781] T 301cab3b867a440591bc6a7e67c380fb P ef432f83c6344df6a96f6774b31c2848 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:39:50.760051 13682 raft_consensus.cc:686] T 301cab3b867a440591bc6a7e67c380fb P ef432f83c6344df6a96f6774b31c2848 [term 1 LEADER]: Becoming Leader. State: Replica: ef432f83c6344df6a96f6774b31c2848, State: Running, Role: LEADER
I20260619 14:39:50.760376 13682 consensus_queue.cc:237] T 301cab3b867a440591bc6a7e67c380fb P ef432f83c6344df6a96f6774b31c2848 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40ccf2a7f1fc48f5a6913e147e2265bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44263 } } peers { permanent_uuid: "ef432f83c6344df6a96f6774b31c2848" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44493 } } peers { permanent_uuid: "ae9f3fcb2c3b464795681d4b4c4179e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44495 } }
I20260619 14:39:50.760864 13462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "301cab3b867a440591bc6a7e67c380fb" candidate_uuid: "ef432f83c6344df6a96f6774b31c2848" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40ccf2a7f1fc48f5a6913e147e2265bb" is_pre_election: true
I20260619 14:39:50.761873 13463 raft_consensus.cc:2454] T 301cab3b867a440591bc6a7e67c380fb P 40ccf2a7f1fc48f5a6913e147e2265bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef432f83c6344df6a96f6774b31c2848 in term 1.
I20260619 14:39:50.763468 13179 catalog_manager.cc:5495] T 301cab3b867a440591bc6a7e67c380fb P ef432f83c6344df6a96f6774b31c2848 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef432f83c6344df6a96f6774b31c2848 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "ef432f83c6344df6a96f6774b31c2848" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40ccf2a7f1fc48f5a6913e147e2265bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef432f83c6344df6a96f6774b31c2848" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44493 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae9f3fcb2c3b464795681d4b4c4179e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44495 } health_report { overall_health: UNKNOWN } } }
I20260619 14:39:50.798573 13676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:50.794055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34600 (local address 127.0.98.1:44493)
0619 14:39:50.794249 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 14:39:50.794254 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:39:50.794267 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:39:50.794347 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:39:50.794350 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:39:50.794411 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:39:50.794506 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:39:50.794791 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:50.795555 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:50.796752 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:50.797084 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:50.797180 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:50.797603 (+ 423us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:39:50.798151 (+ 548us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:39:50.798283 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 14:39:50.798347 (+ 64us) server_negotiation.cc:299] Negotiation successful
0619 14:39:50.798418 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260619 14:39:50.805271 13599 raft_consensus.cc:1261] T 301cab3b867a440591bc6a7e67c380fb P ae9f3fcb2c3b464795681d4b4c4179e7 [term 1 FOLLOWER]: Refusing update from remote peer ef432f83c6344df6a96f6774b31c2848: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:39:50.806001 13682 consensus_queue.cc:1035] T 301cab3b867a440591bc6a7e67c380fb P ef432f83c6344df6a96f6774b31c2848 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae9f3fcb2c3b464795681d4b4c4179e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:39:50.812392 13463 raft_consensus.cc:1261] T 301cab3b867a440591bc6a7e67c380fb P 40ccf2a7f1fc48f5a6913e147e2265bb [term 1 FOLLOWER]: Refusing update from remote peer ef432f83c6344df6a96f6774b31c2848: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:39:50.813143 13682 consensus_queue.cc:1035] T 301cab3b867a440591bc6a7e67c380fb P ef432f83c6344df6a96f6774b31c2848 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40ccf2a7f1fc48f5a6913e147e2265bb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:39:50.817006 13693 mvcc.cc:205] Tried to move back new op lower bound from 7298580442331152384 to 7298580442157207552. Current Snapshot: MvccSnapshot[applied={T|T < 7298580442331152384}]
I20260619 14:39:50.817802 13694 mvcc.cc:205] Tried to move back new op lower bound from 7298580442331152384 to 7298580442157207552. Current Snapshot: MvccSnapshot[applied={T|T < 7298580442331152384}]
Jun 19 14:39:50 dist-test-slave-32hj krb5kdc[12183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 14:39:50.840976 13697 mvcc.cc:205] Tried to move back new op lower bound from 7298580442331152384 to 7298580442157207552. Current Snapshot: MvccSnapshot[applied={T|T < 7298580442331152384 or (T in {7298580442331152384})}]
I20260619 14:39:50.847314 13179 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:39:50.847496 13179 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:39:50.850225 13179 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 08cd4a52f1ad4e4cb0c75d4c328ff80d: Sending DeleteTablet for 3 replicas of tablet 301cab3b867a440591bc6a7e67c380fb
I20260619 14:39:50.851715 13446 tablet_service.cc:1514] Processing DeleteTablet for tablet 301cab3b867a440591bc6a7e67c380fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:39:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54214
I20260619 14:39:50.852229 13311 tablet_service.cc:1514] Processing DeleteTablet for tablet 301cab3b867a440591bc6a7e67c380fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:39:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34584
I20260619 14:39:50.852317 13560 tablet_service.cc:1514] Processing DeleteTablet for tablet 301cab3b867a440591bc6a7e67c380fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:39:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51854
I20260619 14:39:50.852667 13703 tablet_replica.cc:330] T 301cab3b867a440591bc6a7e67c380fb P ef432f83c6344df6a96f6774b31c2848: stopping tablet replica
I20260619 14:39:50.852999 13703 raft_consensus.cc:2229] T 301cab3b867a440591bc6a7e67c380fb P ef432f83c6344df6a96f6774b31c2848 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:39:50.853372 13703 raft_consensus.cc:2258] T 301cab3b867a440591bc6a7e67c380fb P ef432f83c6344df6a96f6774b31c2848 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:39:50.853487 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 13217
I20260619 14:39:50.854189 13704 tablet_replica.cc:330] T 301cab3b867a440591bc6a7e67c380fb P ae9f3fcb2c3b464795681d4b4c4179e7: stopping tablet replica
I20260619 14:39:50.854481 13704 raft_consensus.cc:2229] T 301cab3b867a440591bc6a7e67c380fb P ae9f3fcb2c3b464795681d4b4c4179e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:39:50.854758 13704 raft_consensus.cc:2258] T 301cab3b867a440591bc6a7e67c380fb P ae9f3fcb2c3b464795681d4b4c4179e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:39:50.856118 13704 ts_tablet_manager.cc:1905] T 301cab3b867a440591bc6a7e67c380fb P ae9f3fcb2c3b464795681d4b4c4179e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:39:50.858229 13704 ts_tablet_manager.cc:1918] T 301cab3b867a440591bc6a7e67c380fb P ae9f3fcb2c3b464795681d4b4c4179e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:39:50.858310 13704 log.cc:1196] T 301cab3b867a440591bc6a7e67c380fb P ae9f3fcb2c3b464795681d4b4c4179e7: Deleting WAL directory at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781879837648036-392-0/minicluster-data/ts-2/wal/wals/301cab3b867a440591bc6a7e67c380fb
I20260619 14:39:50.858568 13704 ts_tablet_manager.cc:1939] T 301cab3b867a440591bc6a7e67c380fb P ae9f3fcb2c3b464795681d4b4c4179e7: Deleting consensus metadata
I20260619 14:39:50.859500 13165 catalog_manager.cc:4841] TS ae9f3fcb2c3b464795681d4b4c4179e7 (127.0.98.3:44495): tablet 301cab3b867a440591bc6a7e67c380fb (table test-table [id=e367b1086cfc4c64930ce38558ad5fa4]) successfully deleted
W20260619 14:39:50.860929 13163 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44493 (error 108)
I20260619 14:39:50.861243 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 13381
I20260619 14:39:50.862572 13674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:50.862251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44493 (local address 127.0.0.1:34606)
0619 14:39:50.862453 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:50.862506 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
W20260619 14:39:50.862681 13163 catalog_manager.cc:4584] TS ef432f83c6344df6a96f6774b31c2848 (127.0.98.1:44493): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 301cab3b867a440591bc6a7e67c380fb: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44493: connect: Connection refused (error 111)
I20260619 14:39:50.867018 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 13515
I20260619 14:39:50.869073 13674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:50.868084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44263 (local address 127.0.0.1:54222)
0619 14:39:50.868972 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:50.869016 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:44263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":825}
I20260619 14:39:50.872566 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 13146
I20260619 14:39:50.886458 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 14:39:51.220434 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 14:39:51.220536 392 mini_ranger.cc:66] Stopping Ranger...
I20260619 14:39:51.554436 392 mini_ranger.cc:68] Stopped Ranger
2026-06-19 14:39:51.554 UTC [12213] LOG: received smart shutdown request
2026-06-19 14:39:51.558 UTC [12213] LOG: background worker "logical replication launcher" (PID 12222) exited with exit code 1
2026-06-19 14:39:51.558 UTC [12217] LOG: shutting down
2026-06-19 14:39:51.692 UTC [12213] LOG: database system is shut down
2026-06-19T14:39:51Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (44783 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:39:51 dist-test-slave-32hj krb5kdc[13715](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:39:51 dist-test-slave-32hj krb5kdc[13715](info): set up 2 sockets
Jun 19 14:39:51 dist-test-slave-32hj krb5kdc[13715](info): commencing operation
krb5kdc: starting...
W20260619 14:39:52.736478 392 mini_kdc.cc:121] Time spent starting KDC: real 0.921s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/krb5kdc/test-user.keytab.
Jun 19 14:39:52 dist-test-slave-32hj krb5kdc[13715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:39:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:39:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:39:52.845014 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46841
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46841,127.0.98.61:40169,127.0.98.60:33831
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:39:52.948604 13728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:39:52.948889 13728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:39:52.948931 13728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:39:52.952756 13728 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:39:52.952804 13728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:39:52.952822 13728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:39:52.952837 13728 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:39:52.952855 13728 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:39:52.957741 13728 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46841,127.0.98.61:40169,127.0.98.60:33831
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:39:52.958595 13728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:39:52.959513 13728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:39:52.967473 13728 server_base.cc:1034] running on GCE node
W20260619 14:39:52.967438 13734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:39:52.967463 13733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:39:52.967438 13736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:39:52.968133 13728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:39:52.969136 13728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:39:52.970310 13728 hybrid_clock.cc:648] HybridClock initialized: now 1781879992970298 us; error 25 us; skew 500 ppm
Jun 19 14:39:52 dist-test-slave-32hj krb5kdc[13715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879992, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:39:52.975478 13728 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:39:52.976274 13728 webserver.cc:460] Webserver started at http://127.0.98.62:41595/ using document root <none> and password file <none>
I20260619 14:39:52.976788 13728 fs_manager.cc:359] Metadata directory not provided
I20260619 14:39:52.976851 13728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:39:52.977017 13728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:39:52.978843 13728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/minicluster-data/master-0/data/instance:
uuid: "bd20e113236644298e359ecffd8ef6ba"
format_stamp: "Formatted at 2026-06-19 14:39:52 on dist-test-slave-32hj"
server_key: "ff7fdb77e344159202be84586ba447fb"
server_key_iv: "c330288f3fab1adf40b54e920c7b8379"
server_key_version: "encryptionkey@0"
I20260619 14:39:52.979327 13728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/minicluster-data/master-0/wal/instance:
uuid: "bd20e113236644298e359ecffd8ef6ba"
format_stamp: "Formatted at 2026-06-19 14:39:52 on dist-test-slave-32hj"
server_key: "ff7fdb77e344159202be84586ba447fb"
server_key_iv: "c330288f3fab1adf40b54e920c7b8379"
server_key_version: "encryptionkey@0"
I20260619 14:39:52.982615 13728 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260619 14:39:52.984750 13743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:39:52.985870 13728 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:39:52.985975 13728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/minicluster-data/master-0/wal
uuid: "bd20e113236644298e359ecffd8ef6ba"
format_stamp: "Formatted at 2026-06-19 14:39:52 on dist-test-slave-32hj"
server_key: "ff7fdb77e344159202be84586ba447fb"
server_key_iv: "c330288f3fab1adf40b54e920c7b8379"
server_key_version: "encryptionkey@0"
I20260619 14:39:52.986056 13728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:39:53.001925 13728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:39:53.004873 13728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:39:53.005028 13728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:39:53.011426 13728 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46841
I20260619 14:39:53.011471 13795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46841 every 8 connection(s)
I20260619 14:39:53.012292 13728 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:39:53.014863 13796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:39:53.018376 13796 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46841 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:39:53.019662 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 13728
I20260619 14:39:53.019763 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/minicluster-data/master-0/wal/instance
I20260619 14:39:53.020008 392 external_mini_cluster.cc:1428] Setting key d555f15dc96e3fb82894ae72418e6dd1
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Jun 19 14:39:53 dist-test-slave-32hj krb5kdc[13715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879992, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:39:53.034165 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:53.020571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46841 (local address 127.0.0.1:54238)
0619 14:39:53.021185 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:53.021230 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 14:39:53.022000 (+ 770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:39:53.022621 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:39:53.022626 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:39:53.023026 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:39:53.023249 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:53.023262 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:53.024323 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:53.024326 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:53.025071 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:53.025080 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:53.025661 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:53.025665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:53.025771 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:53.026257 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:39:53.026272 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:39:53.029650 (+ 3378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:39:53.031366 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:39:53.031375 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:39:53.031384 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:39:53.031641 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:39:53.031977 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:39:53.031980 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:39:53.031982 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:39:53.032129 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:39:53.032770 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:39:53.032777 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:39:53.032941 (+ 164us) client_negotiation.cc:769] Sending connection context
0619 14:39:53.033111 (+ 170us) client_negotiation.cc:240] Negotiation successful
0619 14:39:53.033507 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"mutex_wait_us":69,"spinlock_wait_cycles":99712,"thread_start_us":166,"threads_started":1}
I20260619 14:39:53.034164 13798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:53.020571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54238 (local address 127.0.98.62:46841)
0619 14:39:53.021020 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0619 14:39:53.021037 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 14:39:53.021423 (+ 386us) server_negotiation.cc:407] Connection header received
0619 14:39:53.022278 (+ 855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:39:53.022282 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:39:53.022455 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:39:53.022611 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:39:53.023388 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:53.024211 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:53.025173 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:53.025524 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:53.025669 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:53.029782 (+ 4113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:39:53.029787 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:39:53.029796 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:39:53.029816 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:39:53.031269 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:53.031770 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:53.031775 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:53.031776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:53.031889 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:53.032226 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:53.032229 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:53.032230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:53.032636 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:39:53.032771 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 14:39:53.033199 (+ 428us) server_negotiation.cc:299] Negotiation successful
0619 14:39:53.033390 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":146,"threads_started":1}
I20260619 14:39:53.036588 13796 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40169 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:39:53.037256 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:53.037045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40169 (local address 127.0.0.1:55884)
0619 14:39:53.037139 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:53.037202 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260619 14:39:53.037432 13746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40169: connect: Connection refused (error 111)
I20260619 14:39:53.038185 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:53.037974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40169 (local address 127.0.0.1:55886)
0619 14:39:53.038084 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:53.038134 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 14:39:53.038293 13796 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40169: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40169: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260619 14:39:53.042918 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40169
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46841,127.0.98.61:40169,127.0.98.60:33831
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:39:53.087536 13796 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40169 } attempt: 1
I20260619 14:39:53.088371 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:53.088087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40169 (local address 127.0.0.1:55888)
0619 14:39:53.088232 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:53.088292 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 14:39:53.089061 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:53.088852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40169 (local address 127.0.0.1:55892)
0619 14:39:53.088954 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:53.089001 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260619 14:39:53.089175 13796 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40169: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40169: connect: Connection refused (error 111)
W20260619 14:39:53.147656 13803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:39:53.147913 13803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:39:53.147969 13803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:39:53.152037 13803 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:39:53.152099 13803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:39:53.152125 13803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:39:53.152146 13803 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:39:53.152181 13803 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:39:53.157346 13796 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40169 } attempt: 2
I20260619 14:39:53.157513 13803 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46841,127.0.98.61:40169,127.0.98.60:33831
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:39:53.158157 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:53.157874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40169 (local address 127.0.0.1:55906)
0619 14:39:53.158019 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:53.158084 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 14:39:53.158689 13803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:39:53.158921 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:53.158685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40169 (local address 127.0.0.1:55922)
0619 14:39:53.158798 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:53.158861 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 14:39:53.159044 13796 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40169: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40169: connect: Connection refused (error 111)
I20260619 14:39:53.159749 13803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:39:53.167589 13808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:39:53.167590 13811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:39:53.167624 13809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:39:53.167719 13803 server_base.cc:1034] running on GCE node
I20260619 14:39:53.168262 13803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:39:53.169318 13803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:39:53.170506 13803 hybrid_clock.cc:648] HybridClock initialized: now 1781879993170485 us; error 30 us; skew 500 ppm
Jun 19 14:39:53 dist-test-slave-32hj krb5kdc[13715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879993, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:39:53.173102 13803 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260619 14:39:53.173913 13803 webserver.cc:460] Webserver started at http://127.0.98.61:40749/ using document root <none> and password file <none>
I20260619 14:39:53.174427 13803 fs_manager.cc:359] Metadata directory not provided
I20260619 14:39:53.174487 13803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:39:53.174672 13803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:39:53.176585 13803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/minicluster-data/master-1/data/instance:
uuid: "5d20ad3900ef4e96959d4652f033befe"
format_stamp: "Formatted at 2026-06-19 14:39:53 on dist-test-slave-32hj"
server_key: "65a700caceeb4c80abceb94c5246bfd7"
server_key_iv: "baa4aed5683198531ec0506af1988bc6"
server_key_version: "encryptionkey@0"
I20260619 14:39:53.177143 13803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/minicluster-data/master-1/wal/instance:
uuid: "5d20ad3900ef4e96959d4652f033befe"
format_stamp: "Formatted at 2026-06-19 14:39:53 on dist-test-slave-32hj"
server_key: "65a700caceeb4c80abceb94c5246bfd7"
server_key_iv: "baa4aed5683198531ec0506af1988bc6"
server_key_version: "encryptionkey@0"
I20260619 14:39:53.180531 13803 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:39:53.182716 13818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:39:53.183892 13803 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:39:53.184006 13803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/minicluster-data/master-1/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/minicluster-data/master-1/wal
uuid: "5d20ad3900ef4e96959d4652f033befe"
format_stamp: "Formatted at 2026-06-19 14:39:53 on dist-test-slave-32hj"
server_key: "65a700caceeb4c80abceb94c5246bfd7"
server_key_iv: "baa4aed5683198531ec0506af1988bc6"
server_key_version: "encryptionkey@0"
I20260619 14:39:53.184090 13803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:39:53.197183 13803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:39:53.199644 13803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:39:53.199795 13803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:39:53.205925 13803 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:40169
I20260619 14:39:53.205947 13870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:40169 every 8 connection(s)
I20260619 14:39:53.206748 13803 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/minicluster-data/master-1/data/info.pb
I20260619 14:39:53.207816 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 13803
I20260619 14:39:53.207928 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/minicluster-data/master-1/wal/instance
I20260619 14:39:53.208184 392 external_mini_cluster.cc:1428] Setting key 4f8d2ae0e4c166aa81e49366786c95fd
I20260619 14:39:53.209350 13871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:39:53.214031 13871 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46841 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 19 14:39:53 dist-test-slave-32hj krb5kdc[13715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879993, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:39:53.226944 13798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:53.216079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54244 (local address 127.0.98.62:46841)
0619 14:39:53.216221 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 14:39:53.216226 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:39:53.217015 (+ 789us) server_negotiation.cc:407] Connection header received
0619 14:39:53.217859 (+ 844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:39:53.217863 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:39:53.217920 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:39:53.217991 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:39:53.219162 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:53.219672 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:53.220809 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:53.221056 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:53.221141 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:53.223696 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:39:53.223698 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:39:53.223700 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:39:53.223726 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:39:53.225080 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:53.225596 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:53.225599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:53.225600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:53.225678 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:53.226012 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:53.226014 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:53.226016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:53.226188 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:39:53.226260 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 14:39:53.226677 (+ 417us) server_negotiation.cc:299] Negotiation successful
0619 14:39:53.226819 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 14:39:53.227624 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:53.216438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46841 (local address 127.0.0.1:54244)
0619 14:39:53.216879 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:53.216920 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 14:39:53.217675 (+ 755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:39:53.218147 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:39:53.218154 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:39:53.218479 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:39:53.219026 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:53.219039 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:53.219785 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:53.219788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:53.220683 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:53.220693 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:53.221191 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:53.221195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:53.221379 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:53.221931 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:39:53.221951 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:39:53.223507 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:39:53.225187 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:39:53.225193 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:39:53.225207 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:39:53.225454 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:39:53.225793 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:39:53.225795 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:39:53.225796 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:39:53.225936 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:39:53.226258 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:39:53.226262 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:39:53.226470 (+ 208us) client_negotiation.cc:769] Sending connection context
0619 14:39:53.226606 (+ 136us) client_negotiation.cc:240] Negotiation successful
0619 14:39:53.226821 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":116,"threads_started":1}
I20260619 14:39:53.228859 13871 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40169 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260619 14:39:53.231220 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:33831
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46841,127.0.98.61:40169,127.0.98.60:33831
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:39:53 dist-test-slave-32hj krb5kdc[13715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879993, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 14:39:53.237102 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:53.229268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40169 (local address 127.0.0.1:55926)
0619 14:39:53.229392 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:53.229406 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:39:53.229478 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:39:53.229975 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:39:53.229978 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:39:53.230114 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:39:53.230195 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:53.230200 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:53.231099 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:53.231102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:53.231667 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:53.231675 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:53.232185 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:53.232188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:53.232291 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:53.232697 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:39:53.232707 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:39:53.233988 (+ 1281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:39:53.235644 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:39:53.235647 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:39:53.235649 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:39:53.235876 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:39:53.236128 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:39:53.236131 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:39:53.236133 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:39:53.236220 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:39:53.236592 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:39:53.236595 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:39:53.236686 (+ 91us) client_negotiation.cc:769] Sending connection context
0619 14:39:53.236811 (+ 125us) client_negotiation.cc:240] Negotiation successful
0619 14:39:53.236928 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":2}
I20260619 14:39:53.237138 13876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:53.229369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55926 (local address 127.0.98.61:40169)
0619 14:39:53.229641 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0619 14:39:53.229650 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:39:53.229668 (+ 18us) server_negotiation.cc:407] Connection header received
0619 14:39:53.229698 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:39:53.229702 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:39:53.229834 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:39:53.229952 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:39:53.230279 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:53.231007 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:53.231763 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:53.232092 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:53.232179 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:53.234101 (+ 1922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:39:53.234109 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:39:53.234118 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:39:53.234137 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:39:53.235538 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:53.235971 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:53.235975 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:53.235976 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:53.236049 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:53.236301 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:53.236303 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:53.236304 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:53.236505 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:39:53.236597 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 14:39:53.236884 (+ 287us) server_negotiation.cc:299] Negotiation successful
0619 14:39:53.236991 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":186,"thread_start_us":117,"threads_started":1}
I20260619 14:39:53.239065 13871 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33831 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:39:53.239699 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:53.239453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33831 (local address 127.0.0.1:58474)
0619 14:39:53.239561 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:53.239639 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 14:39:53.239884 13822 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33831: connect: Connection refused (error 111)
I20260619 14:39:53.240604 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:53.240394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33831 (local address 127.0.0.1:58476)
0619 14:39:53.240498 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:53.240549 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 14:39:53.240758 13871 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33831: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33831: connect: Connection refused (error 111)
I20260619 14:39:53.250196 13796 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40169 } attempt: 3
Jun 19 14:39:53 dist-test-slave-32hj krb5kdc[13715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879992, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 14:39:53.258944 13876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:53.250723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55928 (local address 127.0.98.61:40169)
0619 14:39:53.250840 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 14:39:53.250843 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:39:53.250856 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:39:53.250986 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:39:53.250988 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:39:53.251024 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:39:53.251087 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:39:53.251954 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:53.252449 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:53.253485 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:53.254118 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:53.254196 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:53.256236 (+ 2040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:39:53.256239 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:39:53.256241 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:39:53.256263 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:39:53.257539 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:53.257949 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:53.257952 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:53.257953 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:53.258019 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:53.258249 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:53.258251 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:53.258253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:53.258410 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:39:53.258478 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 14:39:53.258732 (+ 254us) server_negotiation.cc:299] Negotiation successful
0619 14:39:53.258817 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 14:39:53.258968 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:53.250662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40169 (local address 127.0.0.1:55928)
0619 14:39:53.250798 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:53.250814 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:39:53.250907 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:39:53.251122 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:39:53.251124 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:39:53.251313 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:39:53.251854 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:53.251863 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:53.252544 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:53.252547 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:53.253388 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:53.253399 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:53.254224 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:53.254227 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:53.254330 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:53.254782 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:39:53.254796 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:39:53.256104 (+ 1308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:39:53.257636 (+ 1532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:39:53.257640 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:39:53.257642 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:39:53.257871 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:39:53.258086 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:39:53.258089 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:39:53.258091 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:39:53.258176 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:39:53.258485 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:39:53.258487 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:39:53.258595 (+ 108us) client_negotiation.cc:769] Sending connection context
0619 14:39:53.258689 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 14:39:53.258815 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 14:39:53.259711 13796 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33831 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:39:53.260330 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:53.260134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33831 (local address 127.0.0.1:58482)
0619 14:39:53.260231 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:53.260276 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260619 14:39:53.260941 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:53.260745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33831 (local address 127.0.0.1:58484)
0619 14:39:53.260853 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:53.260895 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260619 14:39:53.261044 13796 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33831: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33831: connect: Connection refused (error 111)
I20260619 14:39:53.290016 13871 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33831 } attempt: 1
I20260619 14:39:53.290962 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:53.290628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33831 (local address 127.0.0.1:58492)
0619 14:39:53.290796 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:53.290869 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 14:39:53.291694 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:53.291450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33831 (local address 127.0.0.1:58498)
0619 14:39:53.291576 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:53.291630 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260619 14:39:53.291822 13871 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33831: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33831: connect: Connection refused (error 111)
I20260619 14:39:53.292174 13796 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33831 } attempt: 1
I20260619 14:39:53.292802 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:53.292522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33831 (local address 127.0.0.1:58512)
0619 14:39:53.292632 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:53.292702 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 14:39:53.293522 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:53.293300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33831 (local address 127.0.0.1:58518)
0619 14:39:53.293423 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:53.293471 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260619 14:39:53.293650 13796 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33831: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33831: connect: Connection refused (error 111)
W20260619 14:39:53.341200 13877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:39:53.341446 13877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:39:53.341491 13877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:39:53.345361 13877 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:39:53.345407 13877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:39:53.345424 13877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:39:53.345439 13877 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:39:53.345467 13877 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:39:53.350523 13877 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46841,127.0.98.61:40169,127.0.98.60:33831
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:33831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:39:53.351462 13877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:39:53.352386 13877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:39:53.359965 13871 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33831 } attempt: 2
W20260619 14:39:53.360131 13883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:39:53.360155 13886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:39:53.360155 13884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:39:53.360342 13877 server_base.cc:1034] running on GCE node
I20260619 14:39:53.360805 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:53.360494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33831 (local address 127.0.0.1:58534)
0619 14:39:53.360637 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:53.360691 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260619 14:39:53.361022 13877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:39:53.361632 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:53.361460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33831 (local address 127.0.0.1:58536)
0619 14:39:53.361536 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:53.361577 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260619 14:39:53.361761 13871 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33831: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33831: connect: Connection refused (error 111)
I20260619 14:39:53.362188 13877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:39:53.363339 13877 hybrid_clock.cc:648] HybridClock initialized: now 1781879993363322 us; error 31 us; skew 500 ppm
Jun 19 14:39:53 dist-test-slave-32hj krb5kdc[13715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879993, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:39:53.365964 13877 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260619 14:39:53.366756 13877 webserver.cc:460] Webserver started at http://127.0.98.60:38257/ using document root <none> and password file <none>
I20260619 14:39:53.367237 13877 fs_manager.cc:359] Metadata directory not provided
I20260619 14:39:53.367307 13877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:39:53.367488 13877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:39:53.368772 13796 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33831 } attempt: 2
I20260619 14:39:53.369379 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:53.369150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33831 (local address 127.0.0.1:58538)
0619 14:39:53.369262 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:53.369310 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 14:39:53.369539 13877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/minicluster-data/master-2/data/instance:
uuid: "d08392c351114353b5ae2a302bee4e42"
format_stamp: "Formatted at 2026-06-19 14:39:53 on dist-test-slave-32hj"
server_key: "700fb7b2eb50089e9a68bdb96b9d07cd"
server_key_iv: "b4f1950b185ab1d1bc65a7ba799590cc"
server_key_version: "encryptionkey@0"
I20260619 14:39:53.369957 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:53.369764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33831 (local address 127.0.0.1:58550)
0619 14:39:53.369855 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:53.369907 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260619 14:39:53.370059 13796 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33831: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33831: connect: Connection refused (error 111)
I20260619 14:39:53.370095 13877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/minicluster-data/master-2/wal/instance:
uuid: "d08392c351114353b5ae2a302bee4e42"
format_stamp: "Formatted at 2026-06-19 14:39:53 on dist-test-slave-32hj"
server_key: "700fb7b2eb50089e9a68bdb96b9d07cd"
server_key_iv: "b4f1950b185ab1d1bc65a7ba799590cc"
server_key_version: "encryptionkey@0"
I20260619 14:39:53.373512 13877 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:39:53.375684 13893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:39:53.376919 13877 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:39:53.377022 13877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/minicluster-data/master-2/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/minicluster-data/master-2/wal
uuid: "d08392c351114353b5ae2a302bee4e42"
format_stamp: "Formatted at 2026-06-19 14:39:53 on dist-test-slave-32hj"
server_key: "700fb7b2eb50089e9a68bdb96b9d07cd"
server_key_iv: "b4f1950b185ab1d1bc65a7ba799590cc"
server_key_version: "encryptionkey@0"
I20260619 14:39:53.377099 13877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:39:53.390204 13877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:39:53.392838 13877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:39:53.392982 13877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:39:53.399221 13877 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:33831
I20260619 14:39:53.399240 13945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:33831 every 8 connection(s)
I20260619 14:39:53.400071 13877 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/minicluster-data/master-2/data/info.pb
I20260619 14:39:53.402626 13946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:39:53.406086 13946 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46841 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:39:53.406517 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 13877
I20260619 14:39:53.406617 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781879837648036-392-0/minicluster-data/master-2/wal/instance
I20260619 14:39:53.406859 392 external_mini_cluster.cc:1428] Setting key 5a259d98c17a22b4b042979341b72de7
Jun 19 14:39:53 dist-test-slave-32hj krb5kdc[13715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879993, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 19 14:39:53 dist-test-slave-32hj krb5kdc[13715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 14:39:53.418413 13798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:53.407851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54252 (local address 127.0.98.62:46841)
0619 14:39:53.407975 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 14:39:53.407979 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:39:53.408548 (+ 569us) server_negotiation.cc:407] Connection header received
0619 14:39:53.409321 (+ 773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:39:53.409325 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:39:53.409382 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:39:53.409469 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:39:53.410803 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:53.411536 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:53.412462 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:53.412670 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:53.412778 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:53.415035 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:39:53.415038 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:39:53.415041 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:39:53.415066 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:39:53.416498 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:53.417037 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:53.417041 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:53.417043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:53.417115 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:53.417422 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:53.417426 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:53.417427 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:53.417595 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:39:53.417677 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 14:39:53.418147 (+ 470us) server_negotiation.cc:299] Negotiation successful
0619 14:39:53.418259 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 14:39:53.419135 13949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:53.407988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46841 (local address 127.0.0.1:54252)
0619 14:39:53.408411 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:53.408455 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 14:39:53.409135 (+ 680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:39:53.410204 (+ 1069us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:39:53.410208 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:39:53.410493 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:39:53.410642 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:53.410651 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:53.411641 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:53.411644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:53.412350 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:53.412359 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:53.412818 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:53.412822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:53.412923 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:53.413437 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:39:53.413452 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:39:53.414900 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:39:53.416609 (+ 1709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:39:53.416617 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:39:53.416626 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:39:53.416952 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:39:53.417195 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:39:53.417198 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:39:53.417200 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:39:53.417319 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:39:53.417689 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:39:53.417695 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:39:53.417931 (+ 236us) client_negotiation.cc:769] Sending connection context
0619 14:39:53.418069 (+ 138us) client_negotiation.cc:240] Negotiation successful
0619 14:39:53.418340 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"spinlock_wait_cycles":1792,"thread_start_us":108,"threads_started":1}
I20260619 14:39:53.419147 13950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:53.407988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58564 (local address 127.0.98.60:33831)
0619 14:39:53.408423 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0619 14:39:53.408436 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:39:53.408466 (+ 30us) server_negotiation.cc:407] Connection header received
0619 14:39:53.409126 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:39:53.409143 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:39:53.409345 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:39:53.409511 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:39:53.410198 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:53.411424 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:53.412211 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:53.412566 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:53.412709 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:53.415438 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:39:53.415444 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:39:53.415457 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:39:53.415481 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:39:53.416976 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:53.417345 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:53.417358 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:53.417359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:53.417428 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:53.417663 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:53.417666 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:53.417668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:53.417904 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:39:53.418066 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 14:39:53.418274 (+ 208us) server_negotiation.cc:299] Negotiation successful
0619 14:39:53.418526 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"spinlock_wait_cycles":36608,"thread_start_us":116,"threads_started":1}
I20260619 14:39:53.420812 13946 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40169 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:39:53 dist-test-slave-32hj krb5kdc[13715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879993, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 19 14:39:53 dist-test-slave-32hj krb5kdc[13715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:39:53.428087 13949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:53.421182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40169 (local address 127.0.0.1:55938)
0619 14:39:53.421263 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:53.421276 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:39:53.421341 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:39:53.421772 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:39:53.421774 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:39:53.421894 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:39:53.421985 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:53.421992 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:53.422708 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:53.422710 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:53.423203 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:53.423212 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:53.423611 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:53.423614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:53.423680 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:53.424159 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:39:53.424168 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:39:53.425284 (+ 1116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:39:53.426741 (+ 1457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:39:53.426744 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:39:53.426746 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:39:53.426937 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:39:53.427188 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:39:53.427190 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:39:53.427191 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:39:53.427254 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:39:53.427638 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:39:53.427641 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:39:53.427735 (+ 94us) client_negotiation.cc:769] Sending connection context
0619 14:39:53.427850 (+ 115us) client_negotiation.cc:240] Negotiation successful
0619 14:39:53.427956 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260619 14:39:53.428130 13876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:53.421433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55938 (local address 127.0.98.61:40169)
0619 14:39:53.421563 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 14:39:53.421568 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:39:53.421582 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:39:53.421622 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:39:53.421625 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:39:53.421670 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:39:53.421754 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:39:53.422093 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:53.422596 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:53.423306 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:53.423508 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:53.423601 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:53.425421 (+ 1820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:39:53.425425 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:39:53.425428 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:39:53.425454 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:39:53.426661 (+ 1207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:53.427022 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:53.427025 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:53.427027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:53.427108 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:53.427324 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:53.427327 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:53.427329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:53.427528 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:39:53.427599 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 14:39:53.427908 (+ 309us) server_negotiation.cc:299] Negotiation successful
0619 14:39:53.428014 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 14:39:53.428828 13946 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33831 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:39:53.429185 13798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:53.421111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54262 (local address 127.0.98.62:46841)
0619 14:39:53.421225 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 14:39:53.421229 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:39:53.421241 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:39:53.421304 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:39:53.421307 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:39:53.421344 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:39:53.421408 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:39:53.422017 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:53.422499 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:53.423230 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:53.423459 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:53.423526 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:53.425951 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:39:53.425954 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:39:53.425956 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:39:53.425975 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:39:53.427385 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:53.427869 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:53.427872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:53.427873 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:53.427950 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:53.428263 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:53.428266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:53.428267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:53.428453 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:39:53.428619 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 14:39:53.428849 (+ 230us) server_negotiation.cc:299] Negotiation successful
0619 14:39:53.428955 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Jun 19 14:39:53 dist-test-slave-32hj krb5kdc[13715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879993, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 19 14:39:53 dist-test-slave-32hj krb5kdc[13715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 14:39:53.437211 13949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:53.429239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33831 (local address 127.0.0.1:58570)
0619 14:39:53.429469 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:53.429482 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:39:53.429560 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:39:53.429761 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:39:53.429764 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:39:53.429895 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:39:53.429959 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:53.429963 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:53.430793 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:53.430796 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:53.431271 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:53.431277 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:53.432102 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:53.432106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:53.432185 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:53.432559 (+ 374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:39:53.432573 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:39:53.433851 (+ 1278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:39:53.435460 (+ 1609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:39:53.435463 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:39:53.435465 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:39:53.435691 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:39:53.435995 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:39:53.435998 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:39:53.436000 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:39:53.436086 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:39:53.436710 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:39:53.436713 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:39:53.436825 (+ 112us) client_negotiation.cc:769] Sending connection context
0619 14:39:53.436919 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 14:39:53.437044 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":4}
I20260619 14:39:53.437213 13950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:53.429284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58570 (local address 127.0.98.60:33831)
0619 14:39:53.429389 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 14:39:53.429392 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:39:53.429545 (+ 153us) server_negotiation.cc:407] Connection header received
0619 14:39:53.429616 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:39:53.429619 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:39:53.429658 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:39:53.429731 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:39:53.430055 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:53.430706 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:53.431364 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:53.432008 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:53.432094 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:53.433977 (+ 1883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:39:53.433981 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:39:53.433984 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:39:53.434009 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:39:53.435347 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:53.435828 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:53.435830 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:53.435831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:53.435904 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:53.436453 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:53.436455 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:53.436456 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:53.436609 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:39:53.436700 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 14:39:53.436977 (+ 277us) server_negotiation.cc:299] Negotiation successful
0619 14:39:53.437096 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":31}
I20260619 14:39:53.437551 13876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:53.430151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55948 (local address 127.0.98.61:40169)
0619 14:39:53.430245 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0619 14:39:53.430249 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:39:53.430333 (+ 84us) server_negotiation.cc:407] Connection header received
0619 14:39:53.430408 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:39:53.430411 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:39:53.430449 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:39:53.430517 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:39:53.430973 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:53.431413 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:53.432077 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:53.432286 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:53.432367 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:53.434621 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:39:53.434625 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:39:53.434627 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:39:53.434651 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:39:53.436089 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:53.436436 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:53.436439 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:53.436440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:53.436508 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:53.436813 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:53.436816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:53.436817 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:53.436980 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:39:53.437195 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0619 14:39:53.437303 (+ 108us) server_negotiation.cc:299] Negotiation successful
0619 14:39:53.437411 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 14:39:53.440534 13946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d08392c351114353b5ae2a302bee4e42: Bootstrap starting.
I20260619 14:39:53.440750 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:39:53.442977 13946 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d08392c351114353b5ae2a302bee4e42: Neither blocks nor log segments found. Creating new log.
I20260619 14:39:53.443889 13946 log.cc:824] T 00000000000000000000000000000000 P d08392c351114353b5ae2a302bee4e42: Log is configured to *not* fsync() on all Append() calls
I20260619 14:39:53.446090 13946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d08392c351114353b5ae2a302bee4e42: No bootstrap required, opened a new log
I20260619 14:39:53.450071 13946 raft_consensus.cc:348] T 00000000000000000000000000000000 P d08392c351114353b5ae2a302bee4e42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd20e113236644298e359ecffd8ef6ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46841 } } peers { permanent_uuid: "5d20ad3900ef4e96959d4652f033befe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40169 } } peers { permanent_uuid: "d08392c351114353b5ae2a302bee4e42" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33831 } }
I20260619 14:39:53.450332 13946 raft_consensus.cc:374] T 00000000000000000000000000000000 P d08392c351114353b5ae2a302bee4e42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:39:53.450389 13946 raft_consensus.cc:729] T 00000000000000000000000000000000 P d08392c351114353b5ae2a302bee4e42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d08392c351114353b5ae2a302bee4e42, State: Initialized, Role: FOLLOWER
I20260619 14:39:53.450891 13946 consensus_queue.cc:260] T 00000000000000000000000000000000 P d08392c351114353b5ae2a302bee4e42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd20e113236644298e359ecffd8ef6ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46841 } } peers { permanent_uuid: "5d20ad3900ef4e96959d4652f033befe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40169 } } peers { permanent_uuid: "d08392c351114353b5ae2a302bee4e42" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33831 } }
I20260619 14:39:53.451412 13959 sys_catalog.cc:455] T 00000000000000000000000000000000 P d08392c351114353b5ae2a302bee4e42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd20e113236644298e359ecffd8ef6ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46841 } } peers { permanent_uuid: "5d20ad3900ef4e96959d4652f033befe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40169 } } peers { permanent_uuid: "d08392c351114353b5ae2a302bee4e42" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33831 } } }
I20260619 14:39:53.451582 13959 sys_catalog.cc:458] T 00000000000000000000000000000000 P d08392c351114353b5ae2a302bee4e42 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:39:53.452108 13950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:53.442603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58582 (local address 127.0.98.60:33831)
0619 14:39:53.442703 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0619 14:39:53.442707 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:39:53.443411 (+ 704us) server_negotiation.cc:407] Connection header received
0619 14:39:53.443547 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:39:53.443550 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:39:53.443590 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:39:53.443633 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:39:53.444364 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:53.445119 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:53.446441 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:53.446772 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:53.446833 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:53.449060 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:39:53.449064 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:39:53.449066 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:39:53.449088 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:39:53.450370 (+ 1282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:53.450730 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:53.450734 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:53.450736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:53.450808 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:53.451050 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:53.451053 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:53.451055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:53.451264 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:39:53.451393 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 14:39:53.451908 (+ 515us) server_negotiation.cc:299] Negotiation successful
0619 14:39:53.452006 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":41}
I20260619 14:39:53.452345 13946 sys_catalog.cc:564] T 00000000000000000000000000000000 P d08392c351114353b5ae2a302bee4e42 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:39:53.453174 13798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:53.442851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54272 (local address 127.0.98.62:46841)
0619 14:39:53.442954 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 14:39:53.442958 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:39:53.443151 (+ 193us) server_negotiation.cc:407] Connection header received
0619 14:39:53.443265 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:39:53.443268 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:39:53.443317 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:39:53.443401 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:39:53.444255 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:53.445008 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:53.446096 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:53.446409 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:53.446486 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:53.448127 (+ 1641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:39:53.448131 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:39:53.448134 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:39:53.448159 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:39:53.450013 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:53.451696 (+ 1683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:53.451699 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:53.451701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:53.451779 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:53.452261 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:53.452265 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:53.452266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:53.452481 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:39:53.452590 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 14:39:53.452950 (+ 360us) server_negotiation.cc:299] Negotiation successful
0619 14:39:53.453062 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 14:39:53.453238 13876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:53.442940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55956 (local address 127.0.98.61:40169)
0619 14:39:53.443041 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 14:39:53.443044 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:39:53.443246 (+ 202us) server_negotiation.cc:407] Connection header received
0619 14:39:53.443414 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:39:53.443418 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:39:53.443460 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:39:53.443506 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:39:53.444298 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:53.445091 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:53.446441 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:53.446764 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:53.446833 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:53.448516 (+ 1683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:39:53.448519 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:39:53.448522 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:39:53.448546 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:39:53.450724 (+ 2178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:53.451501 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:53.451505 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:53.451506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:53.451589 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:53.452294 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:53.452297 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:53.452298 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:53.452511 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:39:53.452622 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 14:39:53.452987 (+ 365us) server_negotiation.cc:299] Negotiation successful
0619 14:39:53.453093 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 14:39:53.453191 13871 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33831 } attempt: 3
Jun 19 14:39:53 dist-test-slave-32hj krb5kdc[13715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879993, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
W20260619 14:39:53.458294 13970 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d08392c351114353b5ae2a302bee4e42: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:39:53.458425 13970 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:39:53.456243 13877 master_runner.cc:186] Error getting master registration for 127.0.98.62:46841: OK, instance_id { permanent_uuid: "bd20e113236644298e359ecffd8ef6ba" instance_seqno: 1781879993009582 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 14:39:53.459513 13877 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 14:39:53.461926 13950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:53.453883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58586 (local address 127.0.98.60:33831)
0619 14:39:53.453974 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0619 14:39:53.453978 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:39:53.453989 (+ 11us) server_negotiation.cc:407] Connection header received
0619 14:39:53.454016 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:39:53.454019 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:39:53.454053 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:39:53.454522 (+ 469us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:39:53.454598 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:53.455113 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:53.455932 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:53.456226 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:53.456329 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:53.458465 (+ 2136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:39:53.458469 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:39:53.458472 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:39:53.458495 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:39:53.460423 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:53.460891 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:53.460894 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:53.460895 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:53.460963 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:53.461220 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:53.461222 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:53.461223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:53.461377 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:39:53.461441 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 14:39:53.461697 (+ 256us) server_negotiation.cc:299] Negotiation successful
0619 14:39:53.461781 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 14:39:53.461956 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:53.453520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33831 (local address 127.0.0.1:58586)
0619 14:39:53.453677 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:53.453690 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:39:53.453756 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:39:53.454171 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:39:53.454174 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:39:53.454331 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:39:53.454421 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:53.454427 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:53.455242 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:53.455245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:53.455819 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:53.455829 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:53.456329 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:53.456333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:53.456417 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:53.456886 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:39:53.456900 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:39:53.458323 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:39:53.460524 (+ 2201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:39:53.460528 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:39:53.460530 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:39:53.460800 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:39:53.461042 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:39:53.461045 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:39:53.461047 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:39:53.461140 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:39:53.461454 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:39:53.461457 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:39:53.461553 (+ 96us) client_negotiation.cc:769] Sending connection context
0619 14:39:53.461660 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 14:39:53.461793 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260619 14:39:53.464947 13871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d20ad3900ef4e96959d4652f033befe: Bootstrap starting.
I20260619 14:39:53.467820 13871 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d20ad3900ef4e96959d4652f033befe: Neither blocks nor log segments found. Creating new log.
I20260619 14:39:53.468755 13871 log.cc:824] T 00000000000000000000000000000000 P 5d20ad3900ef4e96959d4652f033befe: Log is configured to *not* fsync() on all Append() calls
I20260619 14:39:53.469194 13796 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33831 } attempt: 3
I20260619 14:39:53.471875 13871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d20ad3900ef4e96959d4652f033befe: No bootstrap required, opened a new log
Jun 19 14:39:53 dist-test-slave-32hj krb5kdc[13715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879992, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 14:39:53.475322 13871 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d20ad3900ef4e96959d4652f033befe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd20e113236644298e359ecffd8ef6ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46841 } } peers { permanent_uuid: "5d20ad3900ef4e96959d4652f033befe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40169 } } peers { permanent_uuid: "d08392c351114353b5ae2a302bee4e42" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33831 } }
I20260619 14:39:53.475587 13871 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d20ad3900ef4e96959d4652f033befe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:39:53.475677 13871 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d20ad3900ef4e96959d4652f033befe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d20ad3900ef4e96959d4652f033befe, State: Initialized, Role: FOLLOWER
I20260619 14:39:53.476171 13871 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d20ad3900ef4e96959d4652f033befe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd20e113236644298e359ecffd8ef6ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46841 } } peers { permanent_uuid: "5d20ad3900ef4e96959d4652f033befe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40169 } } peers { permanent_uuid: "d08392c351114353b5ae2a302bee4e42" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33831 } }
I20260619 14:39:53.477013 13973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d20ad3900ef4e96959d4652f033befe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd20e113236644298e359ecffd8ef6ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46841 } } peers { permanent_uuid: "5d20ad3900ef4e96959d4652f033befe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40169 } } peers { permanent_uuid: "d08392c351114353b5ae2a302bee4e42" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33831 } } }
I20260619 14:39:53.477444 13973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d20ad3900ef4e96959d4652f033befe [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:39:53.477604 13871 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d20ad3900ef4e96959d4652f033befe [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:39:53.479569 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:53.469626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33831 (local address 127.0.0.1:58592)
0619 14:39:53.469986 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:53.470002 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:39:53.470075 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:39:53.470554 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:39:53.470558 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:39:53.470755 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:39:53.470862 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:53.470869 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:53.471763 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:53.471767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:53.472602 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:53.472612 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:53.473180 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:53.473184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:53.473274 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:53.473797 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:39:53.473811 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:39:53.476148 (+ 2337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:39:53.478039 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:39:53.478043 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:39:53.478045 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:39:53.478286 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:39:53.478583 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:39:53.478586 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:39:53.478588 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:39:53.478670 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:39:53.479081 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:39:53.479084 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:39:53.479176 (+ 92us) client_negotiation.cc:769] Sending connection context
0619 14:39:53.479268 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 14:39:53.479408 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278}
I20260619 14:39:53.480176 13950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:53.470247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58592 (local address 127.0.98.60:33831)
0619 14:39:53.470344 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0619 14:39:53.470348 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:39:53.470360 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:39:53.470394 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:39:53.470396 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:39:53.470436 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:39:53.470515 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:39:53.470981 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:53.471661 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:53.472753 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:53.473069 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:53.473171 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:53.476291 (+ 3120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:39:53.476295 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:39:53.476297 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:39:53.476322 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:39:53.477919 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:53.478404 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:53.478408 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:53.478410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:53.478491 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:53.478753 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:53.478756 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:53.478758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:53.478955 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:39:53.479039 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 14:39:53.479885 (+ 846us) server_negotiation.cc:299] Negotiation successful
0619 14:39:53.479993 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
W20260619 14:39:53.482635 13984 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5d20ad3900ef4e96959d4652f033befe: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:39:53.482710 13984 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:39:53.480602 13803 master_runner.cc:186] Error getting master registration for 127.0.98.62:46841: OK, instance_id { permanent_uuid: "bd20e113236644298e359ecffd8ef6ba" instance_seqno: 1781879993009582 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 14:39:53.484218 13803 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 14:39:53.486760 13796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd20e113236644298e359ecffd8ef6ba: Bootstrap starting.
I20260619 14:39:53.489490 13796 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd20e113236644298e359ecffd8ef6ba: Neither blocks nor log segments found. Creating new log.
I20260619 14:39:53.490397 13796 log.cc:824] T 00000000000000000000000000000000 P bd20e113236644298e359ecffd8ef6ba: Log is configured to *not* fsync() on all Append() calls
I20260619 14:39:53.492579 13796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd20e113236644298e359ecffd8ef6ba: No bootstrap required, opened a new log
I20260619 14:39:53.496590 13796 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd20e113236644298e359ecffd8ef6ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd20e113236644298e359ecffd8ef6ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46841 } } peers { permanent_uuid: "5d20ad3900ef4e96959d4652f033befe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40169 } } peers { permanent_uuid: "d08392c351114353b5ae2a302bee4e42" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33831 } }
I20260619 14:39:53.496909 13796 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd20e113236644298e359ecffd8ef6ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:39:53.497012 13796 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd20e113236644298e359ecffd8ef6ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd20e113236644298e359ecffd8ef6ba, State: Initialized, Role: FOLLOWER
I20260619 14:39:53.497568 13796 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd20e113236644298e359ecffd8ef6ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd20e113236644298e359ecffd8ef6ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46841 } } peers { permanent_uuid: "5d20ad3900ef4e96959d4652f033befe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40169 } } peers { permanent_uuid: "d08392c351114353b5ae2a302bee4e42" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33831 } }
I20260619 14:39:53.498435 13987 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd20e113236644298e359ecffd8ef6ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd20e113236644298e359ecffd8ef6ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46841 } } peers { permanent_uuid: "5d20ad3900ef4e96959d4652f033befe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40169 } } peers { permanent_uuid: "d08392c351114353b5ae2a302bee4e42" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33831 } } }
I20260619 14:39:53.498605 13987 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd20e113236644298e359ecffd8ef6ba [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:39:53.499028 13796 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd20e113236644298e359ecffd8ef6ba [sys.catalog]: configured and running, proceeding with master startup.
W20260619 14:39:53.503912 13998 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bd20e113236644298e359ecffd8ef6ba: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:39:53.504047 13998 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260619 14:39:53.506690 13749 tablet.cc:2369] T 00000000000000000000000000000000 P bd20e113236644298e359ecffd8ef6ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:39:53.508946 13728 master_runner.cc:263] No leader master found from master bd20e113236644298e359ecffd8ef6ba
I20260619 14:39:53.509007 13728 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260619 14:39:53.644134 13899 tablet.cc:2369] T 00000000000000000000000000000000 P d08392c351114353b5ae2a302bee4e42: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:39:53.690430 13987 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd20e113236644298e359ecffd8ef6ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:39:53.690593 13987 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd20e113236644298e359ecffd8ef6ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd20e113236644298e359ecffd8ef6ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46841 } } peers { permanent_uuid: "5d20ad3900ef4e96959d4652f033befe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40169 } } peers { permanent_uuid: "d08392c351114353b5ae2a302bee4e42" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33831 } }
I20260619 14:39:53.691448 13987 leader_election.cc:288] T 00000000000000000000000000000000 P bd20e113236644298e359ecffd8ef6ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d20ad3900ef4e96959d4652f033befe (127.0.98.61:40169), d08392c351114353b5ae2a302bee4e42 (127.0.98.60:33831)
I20260619 14:39:53.691893 13845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bd20e113236644298e359ecffd8ef6ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d20ad3900ef4e96959d4652f033befe" is_pre_election: true
I20260619 14:39:53.691911 13920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bd20e113236644298e359ecffd8ef6ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d08392c351114353b5ae2a302bee4e42" is_pre_election: true
I20260619 14:39:53.692164 13845 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5d20ad3900ef4e96959d4652f033befe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd20e113236644298e359ecffd8ef6ba in term 0.
I20260619 14:39:53.692181 13920 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d08392c351114353b5ae2a302bee4e42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd20e113236644298e359ecffd8ef6ba in term 0.
I20260619 14:39:53.692562 13747 leader_election.cc:302] T 00000000000000000000000000000000 P bd20e113236644298e359ecffd8ef6ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5d20ad3900ef4e96959d4652f033befe, bd20e113236644298e359ecffd8ef6ba, d08392c351114353b5ae2a302bee4e42; no voters:
I20260619 14:39:53.692827 13987 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd20e113236644298e359ecffd8ef6ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:39:53.692912 13987 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd20e113236644298e359ecffd8ef6ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:39:53.692965 13987 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd20e113236644298e359ecffd8ef6ba [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:39:53.693925 13987 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd20e113236644298e359ecffd8ef6ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd20e113236644298e359ecffd8ef6ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46841 } } peers { permanent_uuid: "5d20ad3900ef4e96959d4652f033befe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40169 } } peers { permanent_uuid: "d08392c351114353b5ae2a302bee4e42" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33831 } }
I20260619 14:39:53.694188 13987 leader_election.cc:288] T 00000000000000000000000000000000 P bd20e113236644298e359ecffd8ef6ba [CANDIDATE]: Term 1 election: Requested vote from peers 5d20ad3900ef4e96959d4652f033befe (127.0.98.61:40169), d08392c351114353b5ae2a302bee4e42 (127.0.98.60:33831)
I20260619 14:39:53.694497 13845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bd20e113236644298e359ecffd8ef6ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d20ad3900ef4e96959d4652f033befe"
I20260619 14:39:53.694499 13920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bd20e113236644298e359ecffd8ef6ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d08392c351114353b5ae2a302bee4e42"
I20260619 14:39:53.694603 13845 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d20ad3900ef4e96959d4652f033befe [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:39:53.694604 13920 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d08392c351114353b5ae2a302bee4e42 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:39:53.695536 13920 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d08392c351114353b5ae2a302bee4e42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd20e113236644298e359ecffd8ef6ba in term 1.
I20260619 14:39:53.695547 13845 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5d20ad3900ef4e96959d4652f033befe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd20e113236644298e359ecffd8ef6ba in term 1.
I20260619 14:39:53.695770 13746 leader_election.cc:302] T 00000000000000000000000000000000 P bd20e113236644298e359ecffd8ef6ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d20ad3900ef4e96959d4652f033befe, bd20e113236644298e359ecffd8ef6ba; no voters:
I20260619 14:39:53.695931 13987 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd20e113236644298e359ecffd8ef6ba [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:39:53.696112 13987 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd20e113236644298e359ecffd8ef6ba [term 1 LEADER]: Becoming Leader. State: Replica: bd20e113236644298e359ecffd8ef6ba, State: Running, Role: LEADER
I20260619 14:39:53.696393 13987 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd20e113236644298e359ecffd8ef6ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd20e113236644298e359ecffd8ef6ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46841 } } peers { permanent_uuid: "5d20ad3900ef4e96959d4652f033befe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40169 } } peers { permanent_uuid: "d08392c351114353b5ae2a302bee4e42" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33831 } }
I20260619 14:39:53.698136 14000 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd20e113236644298e359ecffd8ef6ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd20e113236644298e359ecffd8ef6ba. Latest consensus state: current_term: 1 leader_uuid: "bd20e113236644298e359ecffd8ef6ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd20e113236644298e359ecffd8ef6ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46841 } } peers { permanent_uuid: "5d20ad3900ef4e96959d4652f033befe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40169 } } peers { permanent_uuid: "d08392c351114353b5ae2a302bee4e42" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33831 } } }
I20260619 14:39:53.698251 14000 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd20e113236644298e359ecffd8ef6ba [sys.catalog]: This master's current role is: LEADER
I20260619 14:39:53.698578 14002 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:39:53.699404 14002 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260619 14:39:53.701107 13824 tablet.cc:2369] T 00000000000000000000000000000000 P 5d20ad3900ef4e96959d4652f033befe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:39:53.703648 13920 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d08392c351114353b5ae2a302bee4e42 [term 1 FOLLOWER]: Refusing update from remote peer bd20e113236644298e359ecffd8ef6ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:39:53.703774 13845 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5d20ad3900ef4e96959d4652f033befe [term 1 FOLLOWER]: Refusing update from remote peer bd20e113236644298e359ecffd8ef6ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:39:53.704259 14000 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bd20e113236644298e359ecffd8ef6ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "d08392c351114353b5ae2a302bee4e42" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:39:53.704646 14000 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bd20e113236644298e359ecffd8ef6ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d20ad3900ef4e96959d4652f033befe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:39:53.707858 13959 sys_catalog.cc:455] T 00000000000000000000000000000000 P d08392c351114353b5ae2a302bee4e42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd20e113236644298e359ecffd8ef6ba. Latest consensus state: current_term: 1 leader_uuid: "bd20e113236644298e359ecffd8ef6ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd20e113236644298e359ecffd8ef6ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46841 } } peers { permanent_uuid: "5d20ad3900ef4e96959d4652f033befe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40169 } } peers { permanent_uuid: "d08392c351114353b5ae2a302bee4e42" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33831 } } }
I20260619 14:39:53.707885 13973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d20ad3900ef4e96959d4652f033befe [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd20e113236644298e359ecffd8ef6ba. Latest consensus state: current_term: 1 leader_uuid: "bd20e113236644298e359ecffd8ef6ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd20e113236644298e359ecffd8ef6ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46841 } } peers { permanent_uuid: "5d20ad3900ef4e96959d4652f033befe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40169 } } peers { permanent_uuid: "d08392c351114353b5ae2a302bee4e42" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33831 } } }
I20260619 14:39:53.707973 13959 sys_catalog.cc:458] T 00000000000000000000000000000000 P d08392c351114353b5ae2a302bee4e42 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:39:53.707985 13973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d20ad3900ef4e96959d4652f033befe [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:39:53.710106 14000 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd20e113236644298e359ecffd8ef6ba [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bd20e113236644298e359ecffd8ef6ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd20e113236644298e359ecffd8ef6ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46841 } } peers { permanent_uuid: "5d20ad3900ef4e96959d4652f033befe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40169 } } peers { permanent_uuid: "d08392c351114353b5ae2a302bee4e42" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33831 } } }
I20260619 14:39:53.710204 14000 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd20e113236644298e359ecffd8ef6ba [sys.catalog]: This master's current role is: LEADER
I20260619 14:39:53.710280 14003 mvcc.cc:205] Tried to move back new op lower bound from 7298580454204493824 to 7298580454182805504. Current Snapshot: MvccSnapshot[applied={T|T < 7298580454204493824}]
I20260619 14:39:53.710517 13973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d20ad3900ef4e96959d4652f033befe [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bd20e113236644298e359ecffd8ef6ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd20e113236644298e359ecffd8ef6ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46841 } } peers { permanent_uuid: "5d20ad3900ef4e96959d4652f033befe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40169 } } peers { permanent_uuid: "d08392c351114353b5ae2a302bee4e42" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33831 } } }
I20260619 14:39:53.710635 13973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d20ad3900ef4e96959d4652f033befe [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:39:53.711115 14000 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd20e113236644298e359ecffd8ef6ba [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bd20e113236644298e359ecffd8ef6ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd20e113236644298e359ecffd8ef6ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46841 } } peers { permanent_uuid: "5d20ad3900ef4e96959d4652f033befe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40169 } } peers { permanent_uuid: "d08392c351114353b5ae2a302bee4e42" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33831 } } }
I20260619 14:39:53.711215 14000 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd20e113236644298e359ecffd8ef6ba [sys.catalog]: This master's current role is: LEADER
I20260619 14:39:53.711529 14002 catalog_manager.cc:1347] Generated new cluster ID: 68be9b3cae69456698b3bf1954a274dd
I20260619 14:39:53.711582 14002 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:39:53.711975 13959 sys_catalog.cc:455] T 00000000000000000000000000000000 P d08392c351114353b5ae2a302bee4e42 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bd20e113236644298e359ecffd8ef6ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd20e113236644298e359ecffd8ef6ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46841 } } peers { permanent_uuid: "5d20ad3900ef4e96959d4652f033befe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40169 } } peers { permanent_uuid: "d08392c351114353b5ae2a302bee4e42" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33831 } } }
I20260619 14:39:53.712069 13959 sys_catalog.cc:458] T 00000000000000000000000000000000 P d08392c351114353b5ae2a302bee4e42 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:39:53.712046 14004 mvcc.cc:205] Tried to move back new op lower bound from 7298580454204493824 to 7298580454182805504. Current Snapshot: MvccSnapshot[applied={T|T < 7298580454204493824}]
I20260619 14:39:53.719944 14002 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:39:53.721045 14002 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:39:53.731953 14002 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd20e113236644298e359ecffd8ef6ba: Generated new TSK 0
I20260619 14:39:53.732530 14002 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:39:54.459710 13970 catalog_manager.cc:1259] Loaded cluster ID: 68be9b3cae69456698b3bf1954a274dd
I20260619 14:39:54.459790 13970 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d08392c351114353b5ae2a302bee4e42: loading cluster ID for follower catalog manager: success
I20260619 14:39:54.461410 13970 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d08392c351114353b5ae2a302bee4e42: acquiring CA information for follower catalog manager: success
I20260619 14:39:54.462522 13970 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d08392c351114353b5ae2a302bee4e42: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 14:39:54.483944 13984 catalog_manager.cc:1259] Loaded cluster ID: 68be9b3cae69456698b3bf1954a274dd
I20260619 14:39:54.484027 13984 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5d20ad3900ef4e96959d4652f033befe: loading cluster ID for follower catalog manager: success
I20260619 14:39:54.485729 13984 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5d20ad3900ef4e96959d4652f033befe: acquiring CA information for follower catalog manager: success
I20260619 14:39:54.486956 13984 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5d20ad3900ef4e96959d4652f033befe: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 14:39:54.797492 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 13728
I20260619 14:39:54.802707 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 13803
I20260619 14:39:54.807415 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 13877
2026-06-19T14:39:54Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3011 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781879837648036-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:39:54 dist-test-slave-32hj krb5kdc[14018](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:39:54 dist-test-slave-32hj krb5kdc[14018](info): set up 2 sockets
Jun 19 14:39:54 dist-test-slave-32hj krb5kdc[14018](info): commencing operation
krb5kdc: starting...
W20260619 14:39:55.760591 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781879837648036-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781879837648036-392-0/krb5kdc/test-user.keytab.
Jun 19 14:39:55 dist-test-slave-32hj krb5kdc[14018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:39:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:39:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:39:55.868177 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781879837648036-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781879837648036-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781879837648036-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36547
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36547
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:39:55.970222 14031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:39:55.970460 14031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:39:55.970512 14031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:39:55.974380 14031 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:39:55.974428 14031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:39:55.974453 14031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:39:55.974474 14031 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:39:55.974502 14031 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:39:55.979817 14031 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781879837648036-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781879837648036-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781879837648036-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:39:55.980762 14031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:39:55.981678 14031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:39:55.989423 14039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:39:55.989430 14037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:39:55.989465 14031 server_base.cc:1034] running on GCE node
W20260619 14:39:55.989439 14036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:39:55.990195 14031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:39:55.991149 14031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:39:55.992306 14031 hybrid_clock.cc:648] HybridClock initialized: now 1781879995992291 us; error 25 us; skew 500 ppm
Jun 19 14:39:55 dist-test-slave-32hj krb5kdc[14018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879995, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:39:55.994786 14031 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:39:55.995559 14031 webserver.cc:460] Webserver started at http://127.0.98.62:45041/ using document root <none> and password file <none>
I20260619 14:39:55.996028 14031 fs_manager.cc:359] Metadata directory not provided
I20260619 14:39:55.996080 14031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:39:55.996232 14031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:39:55.998061 14031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781879837648036-392-0/minicluster-data/master-0/data/instance:
uuid: "172e29efab0744ad95246223080249ca"
format_stamp: "Formatted at 2026-06-19 14:39:55 on dist-test-slave-32hj"
server_key: "faa10e73bef7f579dd752f8fe4d6d3ed"
server_key_iv: "5b38719735acba40a3bef9bc2e6bf55d"
server_key_version: "encryptionkey@0"
I20260619 14:39:55.998526 14031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781879837648036-392-0/minicluster-data/master-0/wal/instance:
uuid: "172e29efab0744ad95246223080249ca"
format_stamp: "Formatted at 2026-06-19 14:39:55 on dist-test-slave-32hj"
server_key: "faa10e73bef7f579dd752f8fe4d6d3ed"
server_key_iv: "5b38719735acba40a3bef9bc2e6bf55d"
server_key_version: "encryptionkey@0"
I20260619 14:39:56.001804 14031 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:39:56.003896 14046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:39:56.004995 14031 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:39:56.005090 14031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781879837648036-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781879837648036-392-0/minicluster-data/master-0/wal
uuid: "172e29efab0744ad95246223080249ca"
format_stamp: "Formatted at 2026-06-19 14:39:55 on dist-test-slave-32hj"
server_key: "faa10e73bef7f579dd752f8fe4d6d3ed"
server_key_iv: "5b38719735acba40a3bef9bc2e6bf55d"
server_key_version: "encryptionkey@0"
I20260619 14:39:56.005179 14031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781879837648036-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781879837648036-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781879837648036-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:39:56.014173 14031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:39:56.016770 14031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:39:56.016935 14031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:39:56.023180 14031 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36547
I20260619 14:39:56.023195 14098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36547 every 8 connection(s)
I20260619 14:39:56.024102 14031 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:39:56.026661 14099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:39:56.031869 14099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 172e29efab0744ad95246223080249ca: Bootstrap starting.
I20260619 14:39:56.032861 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 14031
I20260619 14:39:56.032956 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781879837648036-392-0/minicluster-data/master-0/wal/instance
I20260619 14:39:56.033258 392 external_mini_cluster.cc:1428] Setting key d08b245994dddf53f75f05a5cefcf9c7
I20260619 14:39:56.034449 14099 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 172e29efab0744ad95246223080249ca: Neither blocks nor log segments found. Creating new log.
I20260619 14:39:56.035141 14099 log.cc:824] T 00000000000000000000000000000000 P 172e29efab0744ad95246223080249ca: Log is configured to *not* fsync() on all Append() calls
I20260619 14:39:56.037200 14099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 172e29efab0744ad95246223080249ca: No bootstrap required, opened a new log
I20260619 14:39:56.041154 14099 raft_consensus.cc:348] T 00000000000000000000000000000000 P 172e29efab0744ad95246223080249ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "172e29efab0744ad95246223080249ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36547 } }
I20260619 14:39:56.041471 14099 raft_consensus.cc:374] T 00000000000000000000000000000000 P 172e29efab0744ad95246223080249ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:39:56.041531 14099 raft_consensus.cc:729] T 00000000000000000000000000000000 P 172e29efab0744ad95246223080249ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 172e29efab0744ad95246223080249ca, State: Initialized, Role: FOLLOWER
Jun 19 14:39:56 dist-test-slave-32hj krb5kdc[14018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:39:56.041929 14099 consensus_queue.cc:260] T 00000000000000000000000000000000 P 172e29efab0744ad95246223080249ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "172e29efab0744ad95246223080249ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36547 } }
I20260619 14:39:56.042038 14099 raft_consensus.cc:388] T 00000000000000000000000000000000 P 172e29efab0744ad95246223080249ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:39:56.042098 14099 raft_consensus.cc:482] T 00000000000000000000000000000000 P 172e29efab0744ad95246223080249ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:39:56.042191 14099 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 172e29efab0744ad95246223080249ca [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:39:56.043082 14099 raft_consensus.cc:504] T 00000000000000000000000000000000 P 172e29efab0744ad95246223080249ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "172e29efab0744ad95246223080249ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36547 } }
I20260619 14:39:56.043432 14099 leader_election.cc:302] T 00000000000000000000000000000000 P 172e29efab0744ad95246223080249ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 172e29efab0744ad95246223080249ca; no voters:
I20260619 14:39:56.043740 14099 leader_election.cc:288] T 00000000000000000000000000000000 P 172e29efab0744ad95246223080249ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:39:56.043845 14104 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 172e29efab0744ad95246223080249ca [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:39:56.044039 14104 raft_consensus.cc:686] T 00000000000000000000000000000000 P 172e29efab0744ad95246223080249ca [term 1 LEADER]: Becoming Leader. State: Replica: 172e29efab0744ad95246223080249ca, State: Running, Role: LEADER
I20260619 14:39:56.044420 14104 consensus_queue.cc:237] T 00000000000000000000000000000000 P 172e29efab0744ad95246223080249ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "172e29efab0744ad95246223080249ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36547 } }
I20260619 14:39:56.044978 14099 sys_catalog.cc:564] T 00000000000000000000000000000000 P 172e29efab0744ad95246223080249ca [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:39:56.045656 14106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 172e29efab0744ad95246223080249ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 172e29efab0744ad95246223080249ca. Latest consensus state: current_term: 1 leader_uuid: "172e29efab0744ad95246223080249ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "172e29efab0744ad95246223080249ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36547 } } }
I20260619 14:39:56.045768 14106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 172e29efab0744ad95246223080249ca [sys.catalog]: This master's current role is: LEADER
I20260619 14:39:56.046396 14105 sys_catalog.cc:455] T 00000000000000000000000000000000 P 172e29efab0744ad95246223080249ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "172e29efab0744ad95246223080249ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "172e29efab0744ad95246223080249ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36547 } } }
I20260619 14:39:56.046491 14105 sys_catalog.cc:458] T 00000000000000000000000000000000 P 172e29efab0744ad95246223080249ca [sys.catalog]: This master's current role is: LEADER
I20260619 14:39:56.046865 14113 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:39:56.048367 14102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:56.034353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60800 (local address 127.0.98.62:36547)
0619 14:39:56.034746 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0619 14:39:56.034761 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:39:56.034796 (+ 35us) server_negotiation.cc:407] Connection header received
0619 14:39:56.035434 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:39:56.035458 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:39:56.035876 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:39:56.036158 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:39:56.036995 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:56.037978 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:56.039143 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:56.039613 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:56.039728 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:56.042570 (+ 2842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:39:56.042575 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:39:56.042599 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:39:56.042625 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:39:56.044475 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:56.045072 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:56.045078 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:56.045083 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:56.045212 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:56.045514 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:56.045517 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:56.045518 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:56.046023 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:39:56.047022 (+ 999us) server_negotiation.cc:1035] Waiting for connection context
0619 14:39:56.047171 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 14:39:56.047457 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":103,"threads_started":1}
I20260619 14:39:56.050216 14113 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:39:56.051577 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:39:56.055183 14113 catalog_manager.cc:1347] Generated new cluster ID: 35d92fe3eedc4791a92364a82f743e33
I20260619 14:39:56.055243 14113 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:39:56.093322 14113 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:39:56.094237 14113 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:39:56.103615 14113 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 172e29efab0744ad95246223080249ca: Generated new TSK 0
I20260619 14:39:56.104388 14113 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:39:56.158200 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 14031
2026-06-19T14:39:56Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1350 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:39:56 dist-test-slave-32hj krb5kdc[14126](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:39:56 dist-test-slave-32hj krb5kdc[14126](info): set up 2 sockets
Jun 19 14:39:56 dist-test-slave-32hj krb5kdc[14126](info): commencing operation
krb5kdc: starting...
W20260619 14:39:57.108613 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/test-user.keytab.
Jun 19 14:39:57 dist-test-slave-32hj krb5kdc[14126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:39:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:39:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:39:57.215603 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42505
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42505
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:39:57.317281 14139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:39:57.317525 14139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:39:57.317564 14139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:39:57.321466 14139 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:39:57.321519 14139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:39:57.321537 14139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:39:57.321555 14139 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:39:57.321583 14139 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:39:57.326692 14139 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42505
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:39:57.327630 14139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:39:57.328554 14139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:39:57.336246 14145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:39:57.336257 14144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:39:57.336344 14139 server_base.cc:1034] running on GCE node
W20260619 14:39:57.336256 14147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:39:57.337009 14139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:39:57.337972 14139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:39:57.339138 14139 hybrid_clock.cc:648] HybridClock initialized: now 1781879997339120 us; error 32 us; skew 500 ppm
Jun 19 14:39:57 dist-test-slave-32hj krb5kdc[14126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879997, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:39:57.341634 14139 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:39:57.342478 14139 webserver.cc:460] Webserver started at http://127.0.98.62:34235/ using document root <none> and password file <none>
I20260619 14:39:57.342975 14139 fs_manager.cc:359] Metadata directory not provided
I20260619 14:39:57.343037 14139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:39:57.343207 14139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:39:57.345109 14139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/master-0/data/instance:
uuid: "2cd8db0aede74336bcb242117ad8d711"
format_stamp: "Formatted at 2026-06-19 14:39:57 on dist-test-slave-32hj"
server_key: "f841e6a7a022fdbbefa013b7c11ded5c"
server_key_iv: "56ae045aede0fd73e4fe6749583233f8"
server_key_version: "encryptionkey@0"
I20260619 14:39:57.345616 14139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/master-0/wal/instance:
uuid: "2cd8db0aede74336bcb242117ad8d711"
format_stamp: "Formatted at 2026-06-19 14:39:57 on dist-test-slave-32hj"
server_key: "f841e6a7a022fdbbefa013b7c11ded5c"
server_key_iv: "56ae045aede0fd73e4fe6749583233f8"
server_key_version: "encryptionkey@0"
I20260619 14:39:57.348929 14139 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:39:57.351094 14154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:39:57.352185 14139 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:39:57.352294 14139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/master-0/wal
uuid: "2cd8db0aede74336bcb242117ad8d711"
format_stamp: "Formatted at 2026-06-19 14:39:57 on dist-test-slave-32hj"
server_key: "f841e6a7a022fdbbefa013b7c11ded5c"
server_key_iv: "56ae045aede0fd73e4fe6749583233f8"
server_key_version: "encryptionkey@0"
I20260619 14:39:57.352378 14139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:39:57.381106 14139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:39:57.383783 14139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:39:57.383947 14139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:39:57.390367 14139 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42505
I20260619 14:39:57.390621 14206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42505 every 8 connection(s)
I20260619 14:39:57.391252 14139 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:39:57.393874 14207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:39:57.399084 14207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711: Bootstrap starting.
I20260619 14:39:57.400465 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 14139
I20260619 14:39:57.400578 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/master-0/wal/instance
I20260619 14:39:57.400858 392 external_mini_cluster.cc:1428] Setting key d26bcc8d8a08d791c58a399deb37c776
I20260619 14:39:57.401521 14207 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711: Neither blocks nor log segments found. Creating new log.
I20260619 14:39:57.402359 14207 log.cc:824] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711: Log is configured to *not* fsync() on all Append() calls
I20260619 14:39:57.404320 14207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711: No bootstrap required, opened a new log
I20260619 14:39:57.408282 14207 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cd8db0aede74336bcb242117ad8d711" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42505 } }
I20260619 14:39:57.408479 14207 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:39:57.408511 14207 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cd8db0aede74336bcb242117ad8d711, State: Initialized, Role: FOLLOWER
I20260619 14:39:57.408926 14207 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cd8db0aede74336bcb242117ad8d711" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42505 } }
I20260619 14:39:57.409050 14207 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:39:57.409114 14207 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:39:57.409209 14207 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711 [term 0 FOLLOWER]: Advancing to term 1
Jun 19 14:39:57 dist-test-slave-32hj krb5kdc[14126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:39:57.410135 14207 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cd8db0aede74336bcb242117ad8d711" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42505 } }
I20260619 14:39:57.410485 14207 leader_election.cc:302] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2cd8db0aede74336bcb242117ad8d711; no voters:
I20260619 14:39:57.410812 14207 leader_election.cc:288] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:39:57.410921 14212 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:39:57.411125 14212 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711 [term 1 LEADER]: Becoming Leader. State: Replica: 2cd8db0aede74336bcb242117ad8d711, State: Running, Role: LEADER
I20260619 14:39:57.411468 14212 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cd8db0aede74336bcb242117ad8d711" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42505 } }
I20260619 14:39:57.412073 14207 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:39:57.412739 14214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2cd8db0aede74336bcb242117ad8d711. Latest consensus state: current_term: 1 leader_uuid: "2cd8db0aede74336bcb242117ad8d711" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cd8db0aede74336bcb242117ad8d711" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42505 } } }
I20260619 14:39:57.413192 14213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2cd8db0aede74336bcb242117ad8d711" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cd8db0aede74336bcb242117ad8d711" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42505 } } }
I20260619 14:39:57.413834 14213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711 [sys.catalog]: This master's current role is: LEADER
I20260619 14:39:57.414062 14214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711 [sys.catalog]: This master's current role is: LEADER
I20260619 14:39:57.414796 14226 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:39:57.415515 14210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:57.401853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45728 (local address 127.0.98.62:42505)
0619 14:39:57.402302 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0619 14:39:57.402320 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 14:39:57.402361 (+ 41us) server_negotiation.cc:407] Connection header received
0619 14:39:57.403088 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:39:57.403110 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:39:57.403498 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:39:57.403833 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:39:57.404454 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:57.405906 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:57.406696 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:57.407426 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:57.407582 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:57.409947 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:39:57.409953 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:39:57.409968 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:39:57.409999 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:39:57.411936 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:57.412438 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:57.412444 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:57.412450 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:57.412584 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:57.413027 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:57.413031 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:57.413033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:57.413557 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:39:57.413767 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0619 14:39:57.414430 (+ 663us) server_negotiation.cc:299] Negotiation successful
0619 14:39:57.414718 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":127,"threads_started":1}
I20260619 14:39:57.417963 14226 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:39:57.423444 14226 catalog_manager.cc:1347] Generated new cluster ID: bc4cc76e8b3a4e5b997014d7858a35b3
I20260619 14:39:57.423506 14226 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:39:57.445041 14226 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:39:57.446334 14226 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:39:57.463160 14226 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711: Generated new TSK 0
I20260619 14:39:57.463995 14226 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 14:39:57.487995 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42505
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:39:57.593885 14233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:39:57.594135 14233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:39:57.594175 14233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:39:57.598089 14233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:39:57.598138 14233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:39:57.598193 14233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:39:57.603292 14233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42505
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:39:57.604231 14233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:39:57.605165 14233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:39:57.614324 14241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:39:57.614447 14233 server_base.cc:1034] running on GCE node
W20260619 14:39:57.614382 14239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:39:57.614354 14238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:39:57.614907 14233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:39:57.615478 14233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:39:57.616627 14233 hybrid_clock.cc:648] HybridClock initialized: now 1781879997616604 us; error 34 us; skew 500 ppm
Jun 19 14:39:57 dist-test-slave-32hj krb5kdc[14126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879997, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:39:57.619205 14233 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:39:57.620011 14233 webserver.cc:460] Webserver started at http://127.0.98.1:39381/ using document root <none> and password file <none>
I20260619 14:39:57.620507 14233 fs_manager.cc:359] Metadata directory not provided
I20260619 14:39:57.620564 14233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:39:57.620811 14233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:39:57.622725 14233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-0/data/instance:
uuid: "79e520561a044d52a83a5b8258842cfc"
format_stamp: "Formatted at 2026-06-19 14:39:57 on dist-test-slave-32hj"
server_key: "7b52a60cbb6be4cac98600c04047f7b6"
server_key_iv: "5e58e9f862ec1996b622c6ed2971d876"
server_key_version: "encryptionkey@0"
I20260619 14:39:57.623236 14233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-0/wal/instance:
uuid: "79e520561a044d52a83a5b8258842cfc"
format_stamp: "Formatted at 2026-06-19 14:39:57 on dist-test-slave-32hj"
server_key: "7b52a60cbb6be4cac98600c04047f7b6"
server_key_iv: "5e58e9f862ec1996b622c6ed2971d876"
server_key_version: "encryptionkey@0"
I20260619 14:39:57.626663 14233 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:39:57.628888 14248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:39:57.630036 14233 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:39:57.630132 14233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-0/wal
uuid: "79e520561a044d52a83a5b8258842cfc"
format_stamp: "Formatted at 2026-06-19 14:39:57 on dist-test-slave-32hj"
server_key: "7b52a60cbb6be4cac98600c04047f7b6"
server_key_iv: "5e58e9f862ec1996b622c6ed2971d876"
server_key_version: "encryptionkey@0"
I20260619 14:39:57.630228 14233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:39:57.642756 14233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:39:57.645711 14233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:39:57.645861 14233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:39:57.646368 14233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:39:57.647106 14233 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:39:57.647151 14233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:39:57.647189 14233 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:39:57.647210 14233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:39:57.655259 14233 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37871
I20260619 14:39:57.655294 14361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37871 every 8 connection(s)
I20260619 14:39:57.656112 14233 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:39:57.663383 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 14233
I20260619 14:39:57.663493 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:39:57.663786 392 external_mini_cluster.cc:1428] Setting key 51788c269141cee0e3ac2aea6a6ddd9c
Jun 19 14:39:57 dist-test-slave-32hj krb5kdc[14126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879997, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:39:57.668828 14210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:57.657752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41709 (local address 127.0.98.62:42505)
0619 14:39:57.657919 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 14:39:57.657924 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:39:57.658713 (+ 789us) server_negotiation.cc:407] Connection header received
0619 14:39:57.659892 (+ 1179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:39:57.659894 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:39:57.659939 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:39:57.660014 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:39:57.661164 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:57.661696 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:57.662735 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:57.662950 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:57.663020 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:57.665369 (+ 2349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:39:57.665373 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:39:57.665375 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:39:57.665403 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:39:57.666778 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:57.667310 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:57.667312 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:57.667313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:57.667381 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:57.667746 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:57.667755 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:57.667756 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:57.667932 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:39:57.668012 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 14:39:57.668561 (+ 549us) server_negotiation.cc:299] Negotiation successful
0619 14:39:57.668653 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 14:39:57.669876 14364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:57.658157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42505 (local address 127.0.98.1:41709)
0619 14:39:57.658567 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:57.658609 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 14:39:57.659712 (+ 1103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:39:57.660194 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:39:57.660201 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:39:57.660537 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:39:57.661051 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:57.661065 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:57.661793 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:57.661795 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:57.662646 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:57.662655 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:57.663052 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:57.663055 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:57.663183 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:57.663735 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:39:57.663755 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:39:57.665224 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:39:57.666891 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:39:57.666897 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:39:57.666911 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:39:57.667227 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:39:57.667455 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:39:57.667457 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:39:57.667459 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:39:57.667625 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:39:57.668034 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:39:57.668040 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:39:57.668341 (+ 301us) client_negotiation.cc:769] Sending connection context
0619 14:39:57.668535 (+ 194us) client_negotiation.cc:240] Negotiation successful
0619 14:39:57.668883 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":112,"threads_started":1}
I20260619 14:39:57.671103 14362 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42505
I20260619 14:39:57.671321 14362 heartbeater.cc:464] Registering TS with master...
I20260619 14:39:57.671746 14362 heartbeater.cc:511] Master 127.0.98.62:42505 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 14:39:57.673156 14171 ts_manager.cc:194] Registered new tserver with Master: 79e520561a044d52a83a5b8258842cfc (127.0.98.1:37871)
I20260619 14:39:57.674261 14171 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41709
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 14:39:57.688138 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42505
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:39:57.793020 14367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:39:57.793294 14367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:39:57.793355 14367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:39:57.797328 14367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:39:57.797391 14367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:39:57.797470 14367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:39:57.802693 14367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42505
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14367
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:39:57.803680 14367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:39:57.804651 14367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:39:57.813640 14372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:39:57.813647 14373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:39:57.813663 14375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:39:57.813800 14367 server_base.cc:1034] running on GCE node
I20260619 14:39:57.814349 14367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:39:57.814924 14367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:39:57.816071 14367 hybrid_clock.cc:648] HybridClock initialized: now 1781879997816052 us; error 31 us; skew 500 ppm
Jun 19 14:39:57 dist-test-slave-32hj krb5kdc[14126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879997, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:39:57.818583 14367 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:39:57.819336 14367 webserver.cc:460] Webserver started at http://127.0.98.2:41345/ using document root <none> and password file <none>
I20260619 14:39:57.819811 14367 fs_manager.cc:359] Metadata directory not provided
I20260619 14:39:57.819865 14367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:39:57.820020 14367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:39:57.821899 14367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-1/data/instance:
uuid: "5769a3dda9664950bd099d906adf485b"
format_stamp: "Formatted at 2026-06-19 14:39:57 on dist-test-slave-32hj"
server_key: "4d8343bc9b01d6896b01a032c9168c6e"
server_key_iv: "1fedf320a17bf400182d154a57caf0d8"
server_key_version: "encryptionkey@0"
I20260619 14:39:57.822368 14367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5769a3dda9664950bd099d906adf485b"
format_stamp: "Formatted at 2026-06-19 14:39:57 on dist-test-slave-32hj"
server_key: "4d8343bc9b01d6896b01a032c9168c6e"
server_key_iv: "1fedf320a17bf400182d154a57caf0d8"
server_key_version: "encryptionkey@0"
I20260619 14:39:57.825704 14367 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:39:57.827831 14382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:39:57.828959 14367 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:39:57.829047 14367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-1/wal
uuid: "5769a3dda9664950bd099d906adf485b"
format_stamp: "Formatted at 2026-06-19 14:39:57 on dist-test-slave-32hj"
server_key: "4d8343bc9b01d6896b01a032c9168c6e"
server_key_iv: "1fedf320a17bf400182d154a57caf0d8"
server_key_version: "encryptionkey@0"
I20260619 14:39:57.829125 14367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:39:57.849750 14367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:39:57.852764 14367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:39:57.852931 14367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:39:57.853430 14367 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:39:57.854140 14367 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:39:57.854182 14367 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:39:57.854218 14367 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:39:57.854238 14367 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:39:57.862151 14367 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41403
I20260619 14:39:57.862190 14495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41403 every 8 connection(s)
I20260619 14:39:57.862973 14367 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:39:57.863190 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 14367
I20260619 14:39:57.863308 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:39:57.863620 392 external_mini_cluster.cc:1428] Setting key 67a96996b12bfca3412b8a18e33ca644
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Jun 19 14:39:57 dist-test-slave-32hj krb5kdc[14126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879997, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 14:39:57.877453 14210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:57.864699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41265 (local address 127.0.98.62:42505)
0619 14:39:57.864861 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 14:39:57.864865 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:39:57.865891 (+ 1026us) server_negotiation.cc:407] Connection header received
0619 14:39:57.867407 (+ 1516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:39:57.867410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:39:57.867463 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:39:57.867532 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:39:57.868941 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:57.869518 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:57.871222 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:57.871443 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:57.871530 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:57.874238 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:39:57.874241 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:39:57.874243 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:39:57.874270 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:39:57.875426 (+ 1156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:57.875917 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:57.875920 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:57.875921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:57.875992 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:57.876369 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:57.876373 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:57.876375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:57.876557 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:39:57.876643 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 14:39:57.877169 (+ 526us) server_negotiation.cc:299] Negotiation successful
0619 14:39:57.877290 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260619 14:39:57.878197 14498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:57.865212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42505 (local address 127.0.98.2:41265)
0619 14:39:57.865736 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:57.865789 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 14:39:57.867202 (+ 1413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:39:57.867724 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:39:57.867731 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:39:57.868178 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:39:57.868806 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:57.868825 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:57.869640 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:57.869644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:57.871102 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:57.871114 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:57.871597 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:57.871602 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:57.871800 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:57.872349 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:39:57.872368 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:39:57.874081 (+ 1713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:39:57.875526 (+ 1445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:39:57.875530 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:39:57.875542 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:39:57.875827 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:39:57.876064 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:39:57.876067 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:39:57.876069 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:39:57.876261 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:39:57.876654 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:39:57.876660 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:39:57.876939 (+ 279us) client_negotiation.cc:769] Sending connection context
0619 14:39:57.877093 (+ 154us) client_negotiation.cc:240] Negotiation successful
0619 14:39:57.877414 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":121,"threads_started":1}
I20260619 14:39:57.879283 14496 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42505
I20260619 14:39:57.879547 14496 heartbeater.cc:464] Registering TS with master...
I20260619 14:39:57.880105 14496 heartbeater.cc:511] Master 127.0.98.62:42505 requested a full tablet report, sending...
I20260619 14:39:57.881244 14171 ts_manager.cc:194] Registered new tserver with Master: 5769a3dda9664950bd099d906adf485b (127.0.98.2:41403)
I20260619 14:39:57.881803 14171 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41265
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 14:39:57.890980 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42505
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:39:57.994552 14501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:39:57.994774 14501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:39:57.994829 14501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:39:57.998967 14501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:39:57.999030 14501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:39:57.999107 14501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:39:58.004199 14501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42505
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:39:58.005215 14501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:39:58.006176 14501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:39:58.015353 14506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:39:58.015348 14507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:39:58.015348 14509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:39:58.015381 14501 server_base.cc:1034] running on GCE node
I20260619 14:39:58.015967 14501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:39:58.016515 14501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:39:58.017688 14501 hybrid_clock.cc:648] HybridClock initialized: now 1781879998017669 us; error 51 us; skew 500 ppm
Jun 19 14:39:58 dist-test-slave-32hj krb5kdc[14126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:39:58.023375 14501 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:39:58.024201 14501 webserver.cc:460] Webserver started at http://127.0.98.3:39511/ using document root <none> and password file <none>
I20260619 14:39:58.024698 14501 fs_manager.cc:359] Metadata directory not provided
I20260619 14:39:58.024796 14501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:39:58.024991 14501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:39:58.026991 14501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-2/data/instance:
uuid: "23b8ca310b6240ad90478bc027bf937c"
format_stamp: "Formatted at 2026-06-19 14:39:58 on dist-test-slave-32hj"
server_key: "9ffa6d0e2ca813649ace55467416ec14"
server_key_iv: "60996965574646544344e5d5d02915a1"
server_key_version: "encryptionkey@0"
I20260619 14:39:58.027503 14501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-2/wal/instance:
uuid: "23b8ca310b6240ad90478bc027bf937c"
format_stamp: "Formatted at 2026-06-19 14:39:58 on dist-test-slave-32hj"
server_key: "9ffa6d0e2ca813649ace55467416ec14"
server_key_iv: "60996965574646544344e5d5d02915a1"
server_key_version: "encryptionkey@0"
I20260619 14:39:58.030988 14501 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:39:58.033352 14516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:39:58.034523 14501 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:39:58.034619 14501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-2/wal
uuid: "23b8ca310b6240ad90478bc027bf937c"
format_stamp: "Formatted at 2026-06-19 14:39:58 on dist-test-slave-32hj"
server_key: "9ffa6d0e2ca813649ace55467416ec14"
server_key_iv: "60996965574646544344e5d5d02915a1"
server_key_version: "encryptionkey@0"
I20260619 14:39:58.034706 14501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:39:58.049253 14501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:39:58.051788 14501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:39:58.051930 14501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:39:58.052426 14501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:39:58.053174 14501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:39:58.053217 14501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:39:58.053251 14501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:39:58.053279 14501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:39:58.061009 14501 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40807
I20260619 14:39:58.061075 14629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40807 every 8 connection(s)
I20260619 14:39:58.061775 14501 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:39:58.066221 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 14501
I20260619 14:39:58.066305 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:39:58.066512 392 external_mini_cluster.cc:1428] Setting key b5d047240682394eb0e47f6c5e3cc63e
Jun 19 14:39:58 dist-test-slave-32hj krb5kdc[14126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:39:58.073699 14210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:58.063334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47557 (local address 127.0.98.62:42505)
0619 14:39:58.063454 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 14:39:58.063458 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:39:58.064188 (+ 730us) server_negotiation.cc:407] Connection header received
0619 14:39:58.065416 (+ 1228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:39:58.065418 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:39:58.065463 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:39:58.065523 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:39:58.066615 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:58.067228 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:58.068380 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:58.068588 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:58.068673 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:58.070760 (+ 2087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:39:58.070763 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:39:58.070765 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:39:58.070787 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:39:58.071874 (+ 1087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:58.072316 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:58.072318 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:58.072320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:58.072395 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:58.072681 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:58.072684 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:58.072686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:58.072899 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:39:58.072985 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 14:39:58.073424 (+ 439us) server_negotiation.cc:299] Negotiation successful
0619 14:39:58.073541 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 14:39:58.074402 14632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:58.063653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42505 (local address 127.0.98.3:47557)
0619 14:39:58.064048 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:58.064092 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 14:39:58.065226 (+ 1134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:39:58.065656 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:39:58.065662 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:39:58.065976 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:39:58.066471 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:58.066485 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:58.067339 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:58.067342 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:58.068269 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:58.068279 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:58.068711 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:58.068739 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:58.068903 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:58.069333 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:39:58.069347 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:39:58.070611 (+ 1264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:39:58.071971 (+ 1360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:39:58.071977 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:39:58.071990 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:39:58.072219 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:39:58.072468 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:39:58.072471 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:39:58.072473 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:39:58.072591 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:39:58.072991 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:39:58.072997 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:39:58.073213 (+ 216us) client_negotiation.cc:769] Sending connection context
0619 14:39:58.073365 (+ 152us) client_negotiation.cc:240] Negotiation successful
0619 14:39:58.073662 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":106,"threads_started":1}
I20260619 14:39:58.075356 14630 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42505
I20260619 14:39:58.075563 14630 heartbeater.cc:464] Registering TS with master...
I20260619 14:39:58.075994 14630 heartbeater.cc:511] Master 127.0.98.62:42505 requested a full tablet report, sending...
I20260619 14:39:58.076910 14171 ts_manager.cc:194] Registered new tserver with Master: 23b8ca310b6240ad90478bc027bf937c (127.0.98.3:40807)
I20260619 14:39:58.077438 14171 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47557
I20260619 14:39:58.079916 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:39:58.087934 14210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:58.081645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57362 (local address 127.0.98.62:42505)
0619 14:39:58.081760 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 14:39:58.081763 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:39:58.081919 (+ 156us) server_negotiation.cc:407] Connection header received
0619 14:39:58.082030 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:39:58.082032 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:39:58.082068 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:39:58.082131 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:39:58.082752 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:58.083233 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:58.084023 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:58.084231 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:58.084312 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:58.085488 (+ 1176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:39:58.085492 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:39:58.085493 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:39:58.085513 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:39:58.086519 (+ 1006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:58.086878 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:58.086882 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:58.086883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:58.086951 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:58.087179 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:58.087183 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:58.087185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:58.087377 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:39:58.087492 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 14:39:58.087699 (+ 207us) server_negotiation.cc:299] Negotiation successful
0619 14:39:58.087800 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 14:39:58.090135 14171 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57362:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:39:58.092973 14171 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:39:58.106035 14645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:58.100444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41403 (local address 127.0.0.1:46342)
0619 14:39:58.101676 (+ 1232us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:58.101688 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:39:58.101828 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:39:58.102273 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:39:58.102276 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:39:58.102291 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:39:58.102372 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:58.102378 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:58.103634 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:58.103637 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:58.104821 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:58.104829 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:58.105546 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:58.105550 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:58.105651 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:58.105686 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 14:39:58.105791 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 14:39:58.105901 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1172,"mutex_wait_us":18,"thread_start_us":85,"threads_started":1}
I20260619 14:39:58.106042 14643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:58.100597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46342 (local address 127.0.98.2:41403)
0619 14:39:58.101020 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0619 14:39:58.101034 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:39:58.101767 (+ 733us) server_negotiation.cc:407] Connection header received
0619 14:39:58.101960 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:39:58.101965 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:39:58.102141 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:39:58.102294 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:39:58.102442 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:58.103513 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:58.104917 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:58.105439 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:58.105533 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:58.105665 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 14:39:58.105827 (+ 162us) server_negotiation.cc:299] Negotiation successful
0619 14:39:58.105911 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":153,"threads_started":1}
I20260619 14:39:58.106192 14641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:58.100329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40807 (local address 127.0.0.1:58492)
0619 14:39:58.101206 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:58.101262 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 14:39:58.101368 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:39:58.102085 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:39:58.102090 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:39:58.102112 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:39:58.102204 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:58.102208 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:58.103322 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:58.103326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:58.104607 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:58.104617 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:58.105812 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:58.105815 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:58.105902 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:58.105923 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 14:39:58.105988 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 14:39:58.106054 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"mutex_wait_us":21,"thread_start_us":76,"threads_started":1}
I20260619 14:39:58.106583 14644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:58.100665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58492 (local address 127.0.98.3:40807)
0619 14:39:58.101631 (+ 966us) server_negotiation.cc:206] Beginning negotiation
0619 14:39:58.101642 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:39:58.101664 (+ 22us) server_negotiation.cc:407] Connection header received
0619 14:39:58.101731 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:39:58.101735 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:39:58.101930 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:39:58.102074 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:39:58.102330 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:58.103217 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:58.104751 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:58.105543 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:58.105633 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:58.105754 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 14:39:58.106356 (+ 602us) server_negotiation.cc:299] Negotiation successful
0619 14:39:58.106448 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":870,"thread_start_us":144,"threads_started":1}
I20260619 14:39:58.107646 14642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:58.101082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37871 (local address 127.0.0.1:57366)
0619 14:39:58.102304 (+ 1222us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:58.102316 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:39:58.102398 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:39:58.102877 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:39:58.102880 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:39:58.102897 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:39:58.102975 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:58.102980 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:58.104173 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:58.104176 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:58.105298 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:58.105308 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:58.107306 (+ 1998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:58.107309 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:58.107391 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:58.107413 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 14:39:58.107470 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 14:39:58.107530 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1162,"mutex_wait_us":22,"thread_start_us":93,"threads_started":1}
I20260619 14:39:58.108151 14646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:58.101322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57366 (local address 127.0.98.1:37871)
0619 14:39:58.102033 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0619 14:39:58.102044 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:39:58.102486 (+ 442us) server_negotiation.cc:407] Connection header received
0619 14:39:58.102560 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:39:58.102565 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:39:58.102737 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:39:58.102886 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:39:58.103065 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:58.104042 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:58.105435 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:58.106488 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:58.106580 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:58.107898 (+ 1318us) server_negotiation.cc:1035] Waiting for connection context
0619 14:39:58.107944 (+ 46us) server_negotiation.cc:299] Negotiation successful
0619 14:39:58.108022 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":123,"threads_started":1}
I20260619 14:39:58.108995 14564 tablet_service.cc:1467] Processing CreateTablet for tablet 3f46adbddc514f05aa3128fbcd953f2c (DEFAULT_TABLE table=test-table [id=83f93c7f07bc433b86d1fa6426a0909a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:39:58.108944 14430 tablet_service.cc:1467] Processing CreateTablet for tablet 3f46adbddc514f05aa3128fbcd953f2c (DEFAULT_TABLE table=test-table [id=83f93c7f07bc433b86d1fa6426a0909a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:39:58.109872 14564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f46adbddc514f05aa3128fbcd953f2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:39:58.109922 14430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f46adbddc514f05aa3128fbcd953f2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:39:58.110575 14296 tablet_service.cc:1467] Processing CreateTablet for tablet 3f46adbddc514f05aa3128fbcd953f2c (DEFAULT_TABLE table=test-table [id=83f93c7f07bc433b86d1fa6426a0909a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:39:58.111488 14296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f46adbddc514f05aa3128fbcd953f2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:39:58.116444 14647 tablet_bootstrap.cc:492] T 3f46adbddc514f05aa3128fbcd953f2c P 23b8ca310b6240ad90478bc027bf937c: Bootstrap starting.
I20260619 14:39:58.117827 14649 tablet_bootstrap.cc:492] T 3f46adbddc514f05aa3128fbcd953f2c P 79e520561a044d52a83a5b8258842cfc: Bootstrap starting.
I20260619 14:39:58.118438 14647 tablet_bootstrap.cc:654] T 3f46adbddc514f05aa3128fbcd953f2c P 23b8ca310b6240ad90478bc027bf937c: Neither blocks nor log segments found. Creating new log.
I20260619 14:39:58.118558 14648 tablet_bootstrap.cc:492] T 3f46adbddc514f05aa3128fbcd953f2c P 5769a3dda9664950bd099d906adf485b: Bootstrap starting.
I20260619 14:39:58.119448 14647 log.cc:824] T 3f46adbddc514f05aa3128fbcd953f2c P 23b8ca310b6240ad90478bc027bf937c: Log is configured to *not* fsync() on all Append() calls
I20260619 14:39:58.120265 14649 tablet_bootstrap.cc:654] T 3f46adbddc514f05aa3128fbcd953f2c P 79e520561a044d52a83a5b8258842cfc: Neither blocks nor log segments found. Creating new log.
I20260619 14:39:58.120987 14648 tablet_bootstrap.cc:654] T 3f46adbddc514f05aa3128fbcd953f2c P 5769a3dda9664950bd099d906adf485b: Neither blocks nor log segments found. Creating new log.
I20260619 14:39:58.121237 14647 tablet_bootstrap.cc:492] T 3f46adbddc514f05aa3128fbcd953f2c P 23b8ca310b6240ad90478bc027bf937c: No bootstrap required, opened a new log
I20260619 14:39:58.121323 14649 log.cc:824] T 3f46adbddc514f05aa3128fbcd953f2c P 79e520561a044d52a83a5b8258842cfc: Log is configured to *not* fsync() on all Append() calls
I20260619 14:39:58.121488 14647 ts_tablet_manager.cc:1397] T 3f46adbddc514f05aa3128fbcd953f2c P 23b8ca310b6240ad90478bc027bf937c: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260619 14:39:58.121874 14648 log.cc:824] T 3f46adbddc514f05aa3128fbcd953f2c P 5769a3dda9664950bd099d906adf485b: Log is configured to *not* fsync() on all Append() calls
I20260619 14:39:58.123266 14649 tablet_bootstrap.cc:492] T 3f46adbddc514f05aa3128fbcd953f2c P 79e520561a044d52a83a5b8258842cfc: No bootstrap required, opened a new log
I20260619 14:39:58.123452 14649 ts_tablet_manager.cc:1397] T 3f46adbddc514f05aa3128fbcd953f2c P 79e520561a044d52a83a5b8258842cfc: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 14:39:58.123667 14648 tablet_bootstrap.cc:492] T 3f46adbddc514f05aa3128fbcd953f2c P 5769a3dda9664950bd099d906adf485b: No bootstrap required, opened a new log
I20260619 14:39:58.123868 14648 ts_tablet_manager.cc:1397] T 3f46adbddc514f05aa3128fbcd953f2c P 5769a3dda9664950bd099d906adf485b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:39:58.124676 14647 raft_consensus.cc:348] T 3f46adbddc514f05aa3128fbcd953f2c P 23b8ca310b6240ad90478bc027bf937c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b8ca310b6240ad90478bc027bf937c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40807 } } peers { permanent_uuid: "5769a3dda9664950bd099d906adf485b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41403 } } peers { permanent_uuid: "79e520561a044d52a83a5b8258842cfc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37871 } }
I20260619 14:39:58.124948 14647 raft_consensus.cc:374] T 3f46adbddc514f05aa3128fbcd953f2c P 23b8ca310b6240ad90478bc027bf937c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:39:58.125018 14647 raft_consensus.cc:729] T 3f46adbddc514f05aa3128fbcd953f2c P 23b8ca310b6240ad90478bc027bf937c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23b8ca310b6240ad90478bc027bf937c, State: Initialized, Role: FOLLOWER
I20260619 14:39:58.125447 14647 consensus_queue.cc:260] T 3f46adbddc514f05aa3128fbcd953f2c P 23b8ca310b6240ad90478bc027bf937c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b8ca310b6240ad90478bc027bf937c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40807 } } peers { permanent_uuid: "5769a3dda9664950bd099d906adf485b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41403 } } peers { permanent_uuid: "79e520561a044d52a83a5b8258842cfc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37871 } }
I20260619 14:39:58.126070 14630 heartbeater.cc:503] Master 127.0.98.62:42505 was elected leader, sending a full tablet report...
I20260619 14:39:58.126336 14647 ts_tablet_manager.cc:1428] T 3f46adbddc514f05aa3128fbcd953f2c P 23b8ca310b6240ad90478bc027bf937c: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:39:58.127498 14648 raft_consensus.cc:348] T 3f46adbddc514f05aa3128fbcd953f2c P 5769a3dda9664950bd099d906adf485b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b8ca310b6240ad90478bc027bf937c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40807 } } peers { permanent_uuid: "5769a3dda9664950bd099d906adf485b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41403 } } peers { permanent_uuid: "79e520561a044d52a83a5b8258842cfc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37871 } }
I20260619 14:39:58.127601 14649 raft_consensus.cc:348] T 3f46adbddc514f05aa3128fbcd953f2c P 79e520561a044d52a83a5b8258842cfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b8ca310b6240ad90478bc027bf937c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40807 } } peers { permanent_uuid: "5769a3dda9664950bd099d906adf485b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41403 } } peers { permanent_uuid: "79e520561a044d52a83a5b8258842cfc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37871 } }
I20260619 14:39:58.127839 14649 raft_consensus.cc:374] T 3f46adbddc514f05aa3128fbcd953f2c P 79e520561a044d52a83a5b8258842cfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:39:58.127846 14648 raft_consensus.cc:374] T 3f46adbddc514f05aa3128fbcd953f2c P 5769a3dda9664950bd099d906adf485b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:39:58.127920 14649 raft_consensus.cc:729] T 3f46adbddc514f05aa3128fbcd953f2c P 79e520561a044d52a83a5b8258842cfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79e520561a044d52a83a5b8258842cfc, State: Initialized, Role: FOLLOWER
I20260619 14:39:58.127920 14648 raft_consensus.cc:729] T 3f46adbddc514f05aa3128fbcd953f2c P 5769a3dda9664950bd099d906adf485b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5769a3dda9664950bd099d906adf485b, State: Initialized, Role: FOLLOWER
I20260619 14:39:58.128382 14648 consensus_queue.cc:260] T 3f46adbddc514f05aa3128fbcd953f2c P 5769a3dda9664950bd099d906adf485b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b8ca310b6240ad90478bc027bf937c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40807 } } peers { permanent_uuid: "5769a3dda9664950bd099d906adf485b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41403 } } peers { permanent_uuid: "79e520561a044d52a83a5b8258842cfc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37871 } }
I20260619 14:39:58.128382 14649 consensus_queue.cc:260] T 3f46adbddc514f05aa3128fbcd953f2c P 79e520561a044d52a83a5b8258842cfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b8ca310b6240ad90478bc027bf937c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40807 } } peers { permanent_uuid: "5769a3dda9664950bd099d906adf485b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41403 } } peers { permanent_uuid: "79e520561a044d52a83a5b8258842cfc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37871 } }
I20260619 14:39:58.128981 14362 heartbeater.cc:503] Master 127.0.98.62:42505 was elected leader, sending a full tablet report...
I20260619 14:39:58.129280 14649 ts_tablet_manager.cc:1428] T 3f46adbddc514f05aa3128fbcd953f2c P 79e520561a044d52a83a5b8258842cfc: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 14:39:58.129832 14648 ts_tablet_manager.cc:1428] T 3f46adbddc514f05aa3128fbcd953f2c P 5769a3dda9664950bd099d906adf485b: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 14:39:58.130136 14496 heartbeater.cc:503] Master 127.0.98.62:42505 was elected leader, sending a full tablet report...
W20260619 14:39:58.157243 14363 tablet.cc:2369] T 3f46adbddc514f05aa3128fbcd953f2c P 79e520561a044d52a83a5b8258842cfc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:39:58.312623 14631 tablet.cc:2369] T 3f46adbddc514f05aa3128fbcd953f2c P 23b8ca310b6240ad90478bc027bf937c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:39:58.364243 14497 tablet.cc:2369] T 3f46adbddc514f05aa3128fbcd953f2c P 5769a3dda9664950bd099d906adf485b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:39:58.502866 14655 raft_consensus.cc:482] T 3f46adbddc514f05aa3128fbcd953f2c P 5769a3dda9664950bd099d906adf485b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:39:58.503098 14655 raft_consensus.cc:504] T 3f46adbddc514f05aa3128fbcd953f2c P 5769a3dda9664950bd099d906adf485b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b8ca310b6240ad90478bc027bf937c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40807 } } peers { permanent_uuid: "5769a3dda9664950bd099d906adf485b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41403 } } peers { permanent_uuid: "79e520561a044d52a83a5b8258842cfc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37871 } }
I20260619 14:39:58.504242 14655 leader_election.cc:288] T 3f46adbddc514f05aa3128fbcd953f2c P 5769a3dda9664950bd099d906adf485b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23b8ca310b6240ad90478bc027bf937c (127.0.98.3:40807), 79e520561a044d52a83a5b8258842cfc (127.0.98.1:37871)
I20260619 14:39:58.508013 14644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:58.504578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57205 (local address 127.0.98.3:40807)
0619 14:39:58.504766 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 14:39:58.504770 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:39:58.504878 (+ 108us) server_negotiation.cc:407] Connection header received
0619 14:39:58.505023 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:39:58.505026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:39:58.505071 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:39:58.505129 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:39:58.505459 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:58.505957 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:58.507090 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:58.507488 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:58.507583 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:58.507645 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 14:39:58.507826 (+ 181us) server_negotiation.cc:299] Negotiation successful
0619 14:39:58.507894 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 14:39:58.508019 14646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:58.504581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39391 (local address 127.0.98.1:37871)
0619 14:39:58.504848 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0619 14:39:58.504853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:39:58.504917 (+ 64us) server_negotiation.cc:407] Connection header received
0619 14:39:58.505036 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:39:58.505038 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:39:58.505079 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:39:58.505147 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:39:58.505441 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:58.505942 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:58.507087 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:58.507465 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:58.507547 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:58.507608 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 14:39:58.507822 (+ 214us) server_negotiation.cc:299] Negotiation successful
0619 14:39:58.507892 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185}
I20260619 14:39:58.508033 14657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:58.504495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37871 (local address 127.0.98.2:39391)
0619 14:39:58.504816 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:58.504828 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:39:58.504938 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:39:58.505168 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:39:58.505170 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:39:58.505184 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:39:58.505334 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:58.505340 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:58.506049 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:58.506053 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:58.506965 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:58.506974 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:58.507578 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:58.507581 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:58.507668 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:58.507720 (+ 52us) client_negotiation.cc:769] Sending connection context
0619 14:39:58.507789 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 14:39:58.507880 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":67,"thread_start_us":92,"threads_started":1}
I20260619 14:39:58.508041 14656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:58.504466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40807 (local address 127.0.98.2:57205)
0619 14:39:58.504779 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:58.504795 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:39:58.504912 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:39:58.505160 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:39:58.505162 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:39:58.505177 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:39:58.505348 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:58.505354 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:58.506060 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:58.506063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:58.506968 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:58.506977 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:58.507603 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:58.507607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:58.507715 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:58.507732 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 14:39:58.507797 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 14:39:58.507897 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":127,"spinlock_wait_cycles":13184,"thread_start_us":69,"threads_started":1}
I20260619 14:39:58.508759 14584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f46adbddc514f05aa3128fbcd953f2c" candidate_uuid: "5769a3dda9664950bd099d906adf485b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b8ca310b6240ad90478bc027bf937c" is_pre_election: true
I20260619 14:39:58.508749 14316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f46adbddc514f05aa3128fbcd953f2c" candidate_uuid: "5769a3dda9664950bd099d906adf485b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79e520561a044d52a83a5b8258842cfc" is_pre_election: true
I20260619 14:39:58.509029 14584 raft_consensus.cc:2454] T 3f46adbddc514f05aa3128fbcd953f2c P 23b8ca310b6240ad90478bc027bf937c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5769a3dda9664950bd099d906adf485b in term 0.
I20260619 14:39:58.509038 14316 raft_consensus.cc:2454] T 3f46adbddc514f05aa3128fbcd953f2c P 79e520561a044d52a83a5b8258842cfc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5769a3dda9664950bd099d906adf485b in term 0.
I20260619 14:39:58.509474 14383 leader_election.cc:302] T 3f46adbddc514f05aa3128fbcd953f2c P 5769a3dda9664950bd099d906adf485b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23b8ca310b6240ad90478bc027bf937c, 5769a3dda9664950bd099d906adf485b; no voters:
I20260619 14:39:58.509718 14655 raft_consensus.cc:2781] T 3f46adbddc514f05aa3128fbcd953f2c P 5769a3dda9664950bd099d906adf485b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:39:58.509836 14655 raft_consensus.cc:482] T 3f46adbddc514f05aa3128fbcd953f2c P 5769a3dda9664950bd099d906adf485b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:39:58.509891 14655 raft_consensus.cc:3037] T 3f46adbddc514f05aa3128fbcd953f2c P 5769a3dda9664950bd099d906adf485b [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:39:58.510855 14655 raft_consensus.cc:504] T 3f46adbddc514f05aa3128fbcd953f2c P 5769a3dda9664950bd099d906adf485b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b8ca310b6240ad90478bc027bf937c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40807 } } peers { permanent_uuid: "5769a3dda9664950bd099d906adf485b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41403 } } peers { permanent_uuid: "79e520561a044d52a83a5b8258842cfc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37871 } }
I20260619 14:39:58.511181 14655 leader_election.cc:288] T 3f46adbddc514f05aa3128fbcd953f2c P 5769a3dda9664950bd099d906adf485b [CANDIDATE]: Term 1 election: Requested vote from peers 23b8ca310b6240ad90478bc027bf937c (127.0.98.3:40807), 79e520561a044d52a83a5b8258842cfc (127.0.98.1:37871)
I20260619 14:39:58.511521 14316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f46adbddc514f05aa3128fbcd953f2c" candidate_uuid: "5769a3dda9664950bd099d906adf485b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79e520561a044d52a83a5b8258842cfc"
I20260619 14:39:58.511514 14584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f46adbddc514f05aa3128fbcd953f2c" candidate_uuid: "5769a3dda9664950bd099d906adf485b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b8ca310b6240ad90478bc027bf937c"
I20260619 14:39:58.511637 14584 raft_consensus.cc:3037] T 3f46adbddc514f05aa3128fbcd953f2c P 23b8ca310b6240ad90478bc027bf937c [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:39:58.511662 14316 raft_consensus.cc:3037] T 3f46adbddc514f05aa3128fbcd953f2c P 79e520561a044d52a83a5b8258842cfc [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:39:58.512899 14316 raft_consensus.cc:2454] T 3f46adbddc514f05aa3128fbcd953f2c P 79e520561a044d52a83a5b8258842cfc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5769a3dda9664950bd099d906adf485b in term 1.
I20260619 14:39:58.512899 14584 raft_consensus.cc:2454] T 3f46adbddc514f05aa3128fbcd953f2c P 23b8ca310b6240ad90478bc027bf937c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5769a3dda9664950bd099d906adf485b in term 1.
I20260619 14:39:58.513175 14383 leader_election.cc:302] T 3f46adbddc514f05aa3128fbcd953f2c P 5769a3dda9664950bd099d906adf485b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23b8ca310b6240ad90478bc027bf937c, 5769a3dda9664950bd099d906adf485b; no voters:
I20260619 14:39:58.513332 14655 raft_consensus.cc:2781] T 3f46adbddc514f05aa3128fbcd953f2c P 5769a3dda9664950bd099d906adf485b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:39:58.513547 14655 raft_consensus.cc:686] T 3f46adbddc514f05aa3128fbcd953f2c P 5769a3dda9664950bd099d906adf485b [term 1 LEADER]: Becoming Leader. State: Replica: 5769a3dda9664950bd099d906adf485b, State: Running, Role: LEADER
I20260619 14:39:58.513883 14655 consensus_queue.cc:237] T 3f46adbddc514f05aa3128fbcd953f2c P 5769a3dda9664950bd099d906adf485b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b8ca310b6240ad90478bc027bf937c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40807 } } peers { permanent_uuid: "5769a3dda9664950bd099d906adf485b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41403 } } peers { permanent_uuid: "79e520561a044d52a83a5b8258842cfc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37871 } }
I20260619 14:39:58.516906 14171 catalog_manager.cc:5495] T 3f46adbddc514f05aa3128fbcd953f2c P 5769a3dda9664950bd099d906adf485b reported cstate change: term changed from 0 to 1, leader changed from <none> to 5769a3dda9664950bd099d906adf485b (127.0.98.2). New cstate: current_term: 1 leader_uuid: "5769a3dda9664950bd099d906adf485b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b8ca310b6240ad90478bc027bf937c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5769a3dda9664950bd099d906adf485b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79e520561a044d52a83a5b8258842cfc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37871 } health_report { overall_health: UNKNOWN } } }
I20260619 14:39:58.553174 14643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:58.549992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46358 (local address 127.0.98.2:41403)
0619 14:39:58.550103 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 14:39:58.550107 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:39:58.550119 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:39:58.550191 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:39:58.550193 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:39:58.550265 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:39:58.550354 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:39:58.550535 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:58.551027 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:58.551885 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:58.552088 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:58.552156 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:58.552392 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:39:58.552844 (+ 452us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:39:58.552933 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 14:39:58.553016 (+ 83us) server_negotiation.cc:299] Negotiation successful
0619 14:39:58.553068 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 14:39:58.558998 14584 raft_consensus.cc:1261] T 3f46adbddc514f05aa3128fbcd953f2c P 23b8ca310b6240ad90478bc027bf937c [term 1 FOLLOWER]: Refusing update from remote peer 5769a3dda9664950bd099d906adf485b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:39:58.558998 14316 raft_consensus.cc:1261] T 3f46adbddc514f05aa3128fbcd953f2c P 79e520561a044d52a83a5b8258842cfc [term 1 FOLLOWER]: Refusing update from remote peer 5769a3dda9664950bd099d906adf485b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:39:58.559686 14658 consensus_queue.cc:1035] T 3f46adbddc514f05aa3128fbcd953f2c P 5769a3dda9664950bd099d906adf485b [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e520561a044d52a83a5b8258842cfc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:39:58.559851 14655 consensus_queue.cc:1035] T 3f46adbddc514f05aa3128fbcd953f2c P 5769a3dda9664950bd099d906adf485b [LEADER]: Connected to new peer: Peer: permanent_uuid: "23b8ca310b6240ad90478bc027bf937c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:39:58.565461 14661 mvcc.cc:205] Tried to move back new op lower bound from 7298580474090778624 to 7298580473915342848. Current Snapshot: MvccSnapshot[applied={T|T < 7298580474090778624}]
I20260619 14:39:58.566509 14663 mvcc.cc:205] Tried to move back new op lower bound from 7298580474090778624 to 7298580473915342848. Current Snapshot: MvccSnapshot[applied={T|T < 7298580474090778624}]
I20260619 14:39:58.573657 14171 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:39:58.573808 14171 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:39:58.575896 14171 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711: Sending DeleteTablet for 3 replicas of tablet 3f46adbddc514f05aa3128fbcd953f2c
I20260619 14:39:58.576537 14564 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f46adbddc514f05aa3128fbcd953f2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:39:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58492
I20260619 14:39:58.576579 14430 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f46adbddc514f05aa3128fbcd953f2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:39:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46342
I20260619 14:39:58.576890 14296 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f46adbddc514f05aa3128fbcd953f2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:39:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57366
I20260619 14:39:58.576978 14672 tablet_replica.cc:330] T 3f46adbddc514f05aa3128fbcd953f2c P 5769a3dda9664950bd099d906adf485b: stopping tablet replica
I20260619 14:39:58.577227 14672 raft_consensus.cc:2229] T 3f46adbddc514f05aa3128fbcd953f2c P 5769a3dda9664950bd099d906adf485b [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:39:58.577328 14673 tablet_replica.cc:330] T 3f46adbddc514f05aa3128fbcd953f2c P 79e520561a044d52a83a5b8258842cfc: stopping tablet replica
I20260619 14:39:58.577503 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 14233
I20260619 14:39:58.577606 14673 raft_consensus.cc:2229] T 3f46adbddc514f05aa3128fbcd953f2c P 79e520561a044d52a83a5b8258842cfc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:39:58.577845 14673 raft_consensus.cc:2258] T 3f46adbddc514f05aa3128fbcd953f2c P 79e520561a044d52a83a5b8258842cfc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:39:58.578733 14672 raft_consensus.cc:2258] T 3f46adbddc514f05aa3128fbcd953f2c P 5769a3dda9664950bd099d906adf485b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:39:58.579903 14672 ts_tablet_manager.cc:1905] T 3f46adbddc514f05aa3128fbcd953f2c P 5769a3dda9664950bd099d906adf485b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:39:58.580911 14671 tablet_replica.cc:330] T 3f46adbddc514f05aa3128fbcd953f2c P 23b8ca310b6240ad90478bc027bf937c: stopping tablet replica
I20260619 14:39:58.581303 14671 raft_consensus.cc:2229] T 3f46adbddc514f05aa3128fbcd953f2c P 23b8ca310b6240ad90478bc027bf937c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:39:58.581687 14671 raft_consensus.cc:2258] T 3f46adbddc514f05aa3128fbcd953f2c P 23b8ca310b6240ad90478bc027bf937c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:39:58.582872 14672 ts_tablet_manager.cc:1918] T 3f46adbddc514f05aa3128fbcd953f2c P 5769a3dda9664950bd099d906adf485b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:39:58.583020 14672 log.cc:1196] T 3f46adbddc514f05aa3128fbcd953f2c P 5769a3dda9664950bd099d906adf485b: Deleting WAL directory at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-1/wal/wals/3f46adbddc514f05aa3128fbcd953f2c
I20260619 14:39:58.583032 14671 ts_tablet_manager.cc:1905] T 3f46adbddc514f05aa3128fbcd953f2c P 23b8ca310b6240ad90478bc027bf937c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:39:58.583405 14672 ts_tablet_manager.cc:1939] T 3f46adbddc514f05aa3128fbcd953f2c P 5769a3dda9664950bd099d906adf485b: Deleting consensus metadata
I20260619 14:39:58.584802 14157 catalog_manager.cc:4841] TS 5769a3dda9664950bd099d906adf485b (127.0.98.2:41403): tablet 3f46adbddc514f05aa3128fbcd953f2c (table test-table [id=83f93c7f07bc433b86d1fa6426a0909a]) successfully deleted
W20260619 14:39:58.585147 14156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37871 (error 108)
I20260619 14:39:58.585408 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 14367
I20260619 14:39:58.585523 14671 ts_tablet_manager.cc:1918] T 3f46adbddc514f05aa3128fbcd953f2c P 23b8ca310b6240ad90478bc027bf937c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:39:58.585601 14671 log.cc:1196] T 3f46adbddc514f05aa3128fbcd953f2c P 23b8ca310b6240ad90478bc027bf937c: Deleting WAL directory at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-2/wal/wals/3f46adbddc514f05aa3128fbcd953f2c
I20260619 14:39:58.585829 14671 ts_tablet_manager.cc:1939] T 3f46adbddc514f05aa3128fbcd953f2c P 23b8ca310b6240ad90478bc027bf937c: Deleting consensus metadata
I20260619 14:39:58.586894 14155 catalog_manager.cc:4841] TS 23b8ca310b6240ad90478bc027bf937c (127.0.98.3:40807): tablet 3f46adbddc514f05aa3128fbcd953f2c (table test-table [id=83f93c7f07bc433b86d1fa6426a0909a]) successfully deleted
I20260619 14:39:58.587411 14642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:58.587158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37871 (local address 127.0.0.1:57380)
0619 14:39:58.587283 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:58.587343 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 14:39:58.587554 14156 catalog_manager.cc:4584] TS 79e520561a044d52a83a5b8258842cfc (127.0.98.1:37871): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3f46adbddc514f05aa3128fbcd953f2c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37871: connect: Connection refused (error 111)
I20260619 14:39:58.590832 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 14501
I20260619 14:39:58.595664 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 14139
I20260619 14:39:58.600620 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42505
--webserver_interface=127.0.98.62
--webserver_port=34235
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42505
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:39:58.703773 14675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:39:58.703998 14675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:39:58.704034 14675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:39:58.707894 14675 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:39:58.707945 14675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:39:58.707962 14675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:39:58.707978 14675 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:39:58.707996 14675 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:39:58.712870 14675 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42505
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=34235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:39:58.713711 14675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:39:58.714586 14675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:39:58.726368 14681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:39:58.726318 14682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:39:58.726385 14675 server_base.cc:1034] running on GCE node
W20260619 14:39:58.726322 14684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:39:58.727207 14675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:39:58.728289 14675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:39:58.729439 14675 hybrid_clock.cc:648] HybridClock initialized: now 1781879998729417 us; error 33 us; skew 500 ppm
Jun 19 14:39:58 dist-test-slave-32hj krb5kdc[14126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:39:58.732038 14675 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:39:58.732897 14675 webserver.cc:460] Webserver started at http://127.0.98.62:34235/ using document root <none> and password file <none>
I20260619 14:39:58.733400 14675 fs_manager.cc:359] Metadata directory not provided
I20260619 14:39:58.733466 14675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:39:58.737057 14675 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260619 14:39:58.738992 14691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:39:58.740129 14675 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:39:58.740222 14675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/master-0/wal
uuid: "2cd8db0aede74336bcb242117ad8d711"
format_stamp: "Formatted at 2026-06-19 14:39:57 on dist-test-slave-32hj"
server_key: "f841e6a7a022fdbbefa013b7c11ded5c"
server_key_iv: "56ae045aede0fd73e4fe6749583233f8"
server_key_version: "encryptionkey@0"
I20260619 14:39:58.740795 14675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:39:58.753432 14675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:39:58.756202 14675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:39:58.756338 14675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:39:58.762765 14675 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42505
I20260619 14:39:58.763000 14743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42505 every 8 connection(s)
I20260619 14:39:58.763772 14675 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:39:58.766103 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 14675
I20260619 14:39:58.766474 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37871
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:42505
--webserver_port=39381
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:39:58.768261 14744 sys_catalog.cc:263] Verifying existing consensus state
I20260619 14:39:58.770396 14744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711: Bootstrap starting.
I20260619 14:39:58.792685 14744 log.cc:824] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711: Log is configured to *not* fsync() on all Append() calls
I20260619 14:39:58.803269 14744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:39:58.803681 14744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711: Bootstrap complete.
I20260619 14:39:58.808912 14744 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cd8db0aede74336bcb242117ad8d711" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42505 } }
I20260619 14:39:58.809397 14744 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cd8db0aede74336bcb242117ad8d711, State: Initialized, Role: FOLLOWER
I20260619 14:39:58.809928 14744 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cd8db0aede74336bcb242117ad8d711" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42505 } }
I20260619 14:39:58.810070 14744 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:39:58.810154 14744 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:39:58.810246 14744 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711 [term 1 FOLLOWER]: Advancing to term 2
I20260619 14:39:58.812502 14744 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cd8db0aede74336bcb242117ad8d711" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42505 } }
I20260619 14:39:58.812887 14744 leader_election.cc:302] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2cd8db0aede74336bcb242117ad8d711; no voters:
I20260619 14:39:58.813215 14744 leader_election.cc:288] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 14:39:58.813334 14748 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 14:39:58.813576 14748 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711 [term 2 LEADER]: Becoming Leader. State: Replica: 2cd8db0aede74336bcb242117ad8d711, State: Running, Role: LEADER
I20260619 14:39:58.813897 14748 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cd8db0aede74336bcb242117ad8d711" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42505 } }
I20260619 14:39:58.814379 14744 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:39:58.815271 14750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2cd8db0aede74336bcb242117ad8d711. Latest consensus state: current_term: 2 leader_uuid: "2cd8db0aede74336bcb242117ad8d711" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cd8db0aede74336bcb242117ad8d711" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42505 } } }
I20260619 14:39:58.815271 14749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2cd8db0aede74336bcb242117ad8d711" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cd8db0aede74336bcb242117ad8d711" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42505 } } }
I20260619 14:39:58.815416 14750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711 [sys.catalog]: This master's current role is: LEADER
I20260619 14:39:58.815416 14749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711 [sys.catalog]: This master's current role is: LEADER
I20260619 14:39:58.815896 14757 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:39:58.820354 14757 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:39:58.821081 14757 catalog_manager.cc:1259] Loaded cluster ID: bc4cc76e8b3a4e5b997014d7858a35b3
I20260619 14:39:58.821139 14757 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:39:58.822479 14757 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260619 14:39:58.827208 14757 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260619 14:39:58.832089 14757 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:39:58.832793 14757 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711: Loaded TSK: 0
I20260619 14:39:58.833911 14757 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 14:39:58.888561 14746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:39:58.888840 14746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:39:58.888912 14746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:39:58.892835 14746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:39:58.892882 14746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:39:58.892956 14746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:39:58.898180 14746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37871
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=39381
--tserver_master_addrs=127.0.98.62:42505
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:39:58.899046 14746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:39:58.899926 14746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:39:58.908531 14774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:39:58.908579 14772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:39:58.908864 14746 server_base.cc:1034] running on GCE node
W20260619 14:39:58.908982 14771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:39:58.909403 14746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:39:58.910108 14746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:39:58.911293 14746 hybrid_clock.cc:648] HybridClock initialized: now 1781879998911269 us; error 30 us; skew 500 ppm
Jun 19 14:39:58 dist-test-slave-32hj krb5kdc[14126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:39:58.914378 14746 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:39:58.915395 14746 webserver.cc:460] Webserver started at http://127.0.98.1:39381/ using document root <none> and password file <none>
I20260619 14:39:58.916015 14746 fs_manager.cc:359] Metadata directory not provided
I20260619 14:39:58.916111 14746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:39:58.920683 14746 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:39:58.923035 14781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:39:58.924144 14746 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:39:58.924247 14746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-0/wal
uuid: "79e520561a044d52a83a5b8258842cfc"
format_stamp: "Formatted at 2026-06-19 14:39:57 on dist-test-slave-32hj"
server_key: "7b52a60cbb6be4cac98600c04047f7b6"
server_key_iv: "5e58e9f862ec1996b622c6ed2971d876"
server_key_version: "encryptionkey@0"
I20260619 14:39:58.924746 14746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:39:58.955129 14746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:39:58.957820 14746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:39:58.957976 14746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:39:58.958559 14746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:39:58.959578 14788 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 14:39:58.963456 14746 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 14:39:58.963510 14746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260619 14:39:58.963559 14746 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 14:39:58.965209 14746 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 14:39:58.965258 14746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260619 14:39:58.965512 14788 tablet_bootstrap.cc:492] T 3f46adbddc514f05aa3128fbcd953f2c P 79e520561a044d52a83a5b8258842cfc: Bootstrap starting.
I20260619 14:39:58.973626 14746 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37871
I20260619 14:39:58.973688 14895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37871 every 8 connection(s)
I20260619 14:39:58.974499 14746 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:39:58.983163 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 14746
I20260619 14:39:58.983610 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:41403
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:42505
--webserver_port=41345
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:39:58.984695 14788 log.cc:824] T 3f46adbddc514f05aa3128fbcd953f2c P 79e520561a044d52a83a5b8258842cfc: Log is configured to *not* fsync() on all Append() calls
Jun 19 14:39:58 dist-test-slave-32hj krb5kdc[14126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:39:58.991933 14899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:58.976622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58457 (local address 127.0.98.62:42505)
0619 14:39:58.977240 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0619 14:39:58.977261 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 14:39:58.977309 (+ 48us) server_negotiation.cc:407] Connection header received
0619 14:39:58.978462 (+ 1153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:39:58.978481 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:39:58.978894 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:39:58.979301 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:39:58.980296 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:58.981956 (+ 1660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:58.983070 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:58.983563 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:58.983712 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:58.986387 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:39:58.986393 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:39:58.986408 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:39:58.986440 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:39:58.988402 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:58.989026 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:58.989032 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:58.989037 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:58.989181 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:58.989556 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:58.989559 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:58.989561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:58.990001 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:39:58.990131 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 14:39:58.990836 (+ 705us) server_negotiation.cc:299] Negotiation successful
0619 14:39:58.991122 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":171,"threads_started":1}
I20260619 14:39:58.992424 14898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:58.976561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42505 (local address 127.0.98.1:58457)
0619 14:39:58.977034 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:58.977085 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 14:39:58.977877 (+ 792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:39:58.979330 (+ 1453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:39:58.979337 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:39:58.979665 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:39:58.980132 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:58.980145 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:58.982093 (+ 1948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:58.982095 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:58.982952 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:58.982961 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:58.983885 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:58.983890 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:58.984031 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:58.984565 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:39:58.984581 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:39:58.986202 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:39:58.988528 (+ 2326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:39:58.988535 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:39:58.988550 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:39:58.988916 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:39:58.989275 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:39:58.989278 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:39:58.989280 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:39:58.989453 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:39:58.990124 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:39:58.990131 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:39:58.990431 (+ 300us) client_negotiation.cc:769] Sending connection context
0619 14:39:58.990677 (+ 246us) client_negotiation.cc:240] Negotiation successful
0619 14:39:58.991022 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":137,"threads_started":1}
I20260619 14:39:58.992761 14788 tablet_bootstrap.cc:492] T 3f46adbddc514f05aa3128fbcd953f2c P 79e520561a044d52a83a5b8258842cfc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:39:58.993134 14788 tablet_bootstrap.cc:492] T 3f46adbddc514f05aa3128fbcd953f2c P 79e520561a044d52a83a5b8258842cfc: Bootstrap complete.
I20260619 14:39:58.994172 14788 ts_tablet_manager.cc:1397] T 3f46adbddc514f05aa3128fbcd953f2c P 79e520561a044d52a83a5b8258842cfc: Time spent bootstrapping tablet: real 0.029s user 0.011s sys 0.016s
I20260619 14:39:58.994900 14896 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42505
I20260619 14:39:58.995149 14896 heartbeater.cc:464] Registering TS with master...
I20260619 14:39:58.995683 14896 heartbeater.cc:511] Master 127.0.98.62:42505 requested a full tablet report, sending...
I20260619 14:39:58.996681 14788 raft_consensus.cc:348] T 3f46adbddc514f05aa3128fbcd953f2c P 79e520561a044d52a83a5b8258842cfc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b8ca310b6240ad90478bc027bf937c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40807 } } peers { permanent_uuid: "5769a3dda9664950bd099d906adf485b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41403 } } peers { permanent_uuid: "79e520561a044d52a83a5b8258842cfc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37871 } }
I20260619 14:39:58.997216 14788 raft_consensus.cc:729] T 3f46adbddc514f05aa3128fbcd953f2c P 79e520561a044d52a83a5b8258842cfc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79e520561a044d52a83a5b8258842cfc, State: Initialized, Role: FOLLOWER
I20260619 14:39:58.997700 14788 consensus_queue.cc:260] T 3f46adbddc514f05aa3128fbcd953f2c P 79e520561a044d52a83a5b8258842cfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b8ca310b6240ad90478bc027bf937c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40807 } } peers { permanent_uuid: "5769a3dda9664950bd099d906adf485b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41403 } } peers { permanent_uuid: "79e520561a044d52a83a5b8258842cfc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37871 } }
I20260619 14:39:58.998703 14788 ts_tablet_manager.cc:1428] T 3f46adbddc514f05aa3128fbcd953f2c P 79e520561a044d52a83a5b8258842cfc: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260619 14:39:58.999788 14708 ts_manager.cc:194] Registered new tserver with Master: 79e520561a044d52a83a5b8258842cfc (127.0.98.1:37871)
I20260619 14:39:59.003480 14708 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58457
I20260619 14:39:59.005992 14896 heartbeater.cc:503] Master 127.0.98.62:42505 was elected leader, sending a full tablet report...
I20260619 14:39:59.009095 14904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:59.002950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57390 (local address 127.0.98.1:37871)
0619 14:39:59.004924 (+ 1974us) server_negotiation.cc:206] Beginning negotiation
0619 14:39:59.004934 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:39:59.004951 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:39:59.005001 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:39:59.005005 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:39:59.005167 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:39:59.005300 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:39:59.005558 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:59.006269 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:59.007982 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:59.008486 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:59.008569 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:59.008659 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 14:39:59.008917 (+ 258us) server_negotiation.cc:299] Negotiation successful
0619 14:39:59.008994 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1887,"thread_start_us":104,"threads_started":1}
I20260619 14:39:59.009233 14903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:59.002868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37871 (local address 127.0.0.1:57390)
0619 14:39:59.003414 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:59.003464 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 14:39:59.003584 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:39:59.005320 (+ 1736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:39:59.005325 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:39:59.005348 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:39:59.005459 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:59.005465 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:59.006385 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:59.006388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:59.007858 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:59.007870 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:59.008595 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:59.008598 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:59.008715 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:59.008785 (+ 70us) client_negotiation.cc:769] Sending connection context
0619 14:39:59.008891 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 14:39:59.009011 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":86,"threads_started":1}
I20260619 14:39:59.010157 14830 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f46adbddc514f05aa3128fbcd953f2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:39:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57390
I20260619 14:39:59.011055 14905 tablet_replica.cc:330] stopping tablet replica
I20260619 14:39:59.011332 14905 raft_consensus.cc:2229] T 3f46adbddc514f05aa3128fbcd953f2c P 79e520561a044d52a83a5b8258842cfc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:39:59.011540 14905 raft_consensus.cc:2258] T 3f46adbddc514f05aa3128fbcd953f2c P 79e520561a044d52a83a5b8258842cfc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:39:59.012399 14905 ts_tablet_manager.cc:1905] T 3f46adbddc514f05aa3128fbcd953f2c P 79e520561a044d52a83a5b8258842cfc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:39:59.015918 14905 ts_tablet_manager.cc:1918] T 3f46adbddc514f05aa3128fbcd953f2c P 79e520561a044d52a83a5b8258842cfc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:39:59.016000 14905 log.cc:1196] T 3f46adbddc514f05aa3128fbcd953f2c P 79e520561a044d52a83a5b8258842cfc: Deleting WAL directory at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-0/wal/wals/3f46adbddc514f05aa3128fbcd953f2c
I20260619 14:39:59.016214 14905 ts_tablet_manager.cc:1939] T 3f46adbddc514f05aa3128fbcd953f2c P 79e520561a044d52a83a5b8258842cfc: Deleting consensus metadata
I20260619 14:39:59.017805 14693 catalog_manager.cc:4841] TS 79e520561a044d52a83a5b8258842cfc (127.0.98.1:37871): tablet 3f46adbddc514f05aa3128fbcd953f2c (table test-table [id=83f93c7f07bc433b86d1fa6426a0909a]) successfully deleted
W20260619 14:39:59.099805 14900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:39:59.100056 14900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:39:59.100102 14900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:39:59.104272 14900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:39:59.104338 14900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:39:59.104398 14900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:39:59.109562 14900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:41403
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=41345
--tserver_master_addrs=127.0.98.62:42505
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:39:59.110666 14900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:39:59.111631 14900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:39:59.121064 14913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:39:59.121069 14910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:39:59.121069 14911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:39:59.121292 14900 server_base.cc:1034] running on GCE node
I20260619 14:39:59.121686 14900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:39:59.122298 14900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:39:59.123467 14900 hybrid_clock.cc:648] HybridClock initialized: now 1781879999123452 us; error 30 us; skew 500 ppm
Jun 19 14:39:59 dist-test-slave-32hj krb5kdc[14126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:39:59.126385 14900 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:39:59.127242 14900 webserver.cc:460] Webserver started at http://127.0.98.2:41345/ using document root <none> and password file <none>
I20260619 14:39:59.127732 14900 fs_manager.cc:359] Metadata directory not provided
I20260619 14:39:59.127797 14900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:39:59.131556 14900 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:39:59.133496 14920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:39:59.134683 14900 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:39:59.134786 14900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-1/wal
uuid: "5769a3dda9664950bd099d906adf485b"
format_stamp: "Formatted at 2026-06-19 14:39:57 on dist-test-slave-32hj"
server_key: "4d8343bc9b01d6896b01a032c9168c6e"
server_key_iv: "1fedf320a17bf400182d154a57caf0d8"
server_key_version: "encryptionkey@0"
I20260619 14:39:59.135317 14900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:39:59.160657 14900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:39:59.163537 14900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:39:59.163715 14900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:39:59.164306 14900 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:39:59.165172 14900 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:39:59.165221 14900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:39:59.165271 14900 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:39:59.165305 14900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:39:59.174238 14900 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41403
I20260619 14:39:59.174265 15033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41403 every 8 connection(s)
I20260619 14:39:59.175235 14900 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:39:59.180001 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 14900
I20260619 14:39:59.180450 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:40807
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:42505
--webserver_port=39511
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:39:59 dist-test-slave-32hj krb5kdc[14126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:39:59.188426 14899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:59.176994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54399 (local address 127.0.98.62:42505)
0619 14:39:59.177154 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 14:39:59.177159 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:39:59.177944 (+ 785us) server_negotiation.cc:407] Connection header received
0619 14:39:59.179142 (+ 1198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:39:59.179144 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:39:59.179195 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:39:59.179266 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:39:59.180585 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:59.181320 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:59.182636 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:59.182977 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:59.183064 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:59.185312 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:39:59.185315 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:39:59.185317 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:39:59.185342 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:39:59.186614 (+ 1272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:59.187060 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:59.187063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:59.187065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:59.187139 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:59.187430 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:59.187433 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:59.187435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:59.187599 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:39:59.187682 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 14:39:59.188129 (+ 447us) server_negotiation.cc:299] Negotiation successful
0619 14:39:59.188263 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 14:39:59.189204 15036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:59.177385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42505 (local address 127.0.98.2:54399)
0619 14:39:59.177796 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:59.177838 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 14:39:59.178955 (+ 1117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:39:59.179407 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:39:59.179413 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:39:59.179748 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:39:59.180408 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:59.180428 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:59.181453 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:59.181456 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:59.182507 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:59.182517 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:59.183106 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:59.183109 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:59.183254 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:59.183804 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:39:59.183819 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:39:59.185155 (+ 1336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:39:59.186715 (+ 1560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:39:59.186727 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:39:59.186741 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:39:59.186970 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:39:59.187218 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:39:59.187221 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:39:59.187223 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:39:59.187348 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:39:59.187693 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:39:59.187699 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:39:59.187919 (+ 220us) client_negotiation.cc:769] Sending connection context
0619 14:39:59.188073 (+ 154us) client_negotiation.cc:240] Negotiation successful
0619 14:39:59.188374 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":100,"threads_started":1}
I20260619 14:39:59.190261 15034 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42505
I20260619 14:39:59.190467 15034 heartbeater.cc:464] Registering TS with master...
I20260619 14:39:59.190894 15034 heartbeater.cc:511] Master 127.0.98.62:42505 requested a full tablet report, sending...
I20260619 14:39:59.191881 14708 ts_manager.cc:194] Registered new tserver with Master: 5769a3dda9664950bd099d906adf485b (127.0.98.2:41403)
I20260619 14:39:59.192584 14708 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54399
W20260619 14:39:59.285956 15037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:39:59.286176 15037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:39:59.286222 15037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:39:59.290045 15037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:39:59.290091 15037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:39:59.290144 15037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:39:59.295105 15037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:40807
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=39511
--tserver_master_addrs=127.0.98.62:42505
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:39:59.296067 15037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:39:59.297016 15037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:39:59.305848 15042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:39:59.305912 15037 server_base.cc:1034] running on GCE node
W20260619 14:39:59.305853 15043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:39:59.305855 15045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:39:59.306464 15037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:39:59.307019 15037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:39:59.308185 15037 hybrid_clock.cc:648] HybridClock initialized: now 1781879999308145 us; error 51 us; skew 500 ppm
Jun 19 14:39:59 dist-test-slave-32hj krb5kdc[14126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:39:59.310663 15037 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:39:59.311405 15037 webserver.cc:460] Webserver started at http://127.0.98.3:39511/ using document root <none> and password file <none>
I20260619 14:39:59.311887 15037 fs_manager.cc:359] Metadata directory not provided
I20260619 14:39:59.311959 15037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:39:59.315630 15037 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260619 14:39:59.317629 15052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:39:59.318754 15037 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:39:59.318846 15037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-2/wal
uuid: "23b8ca310b6240ad90478bc027bf937c"
format_stamp: "Formatted at 2026-06-19 14:39:58 on dist-test-slave-32hj"
server_key: "9ffa6d0e2ca813649ace55467416ec14"
server_key_iv: "60996965574646544344e5d5d02915a1"
server_key_version: "encryptionkey@0"
I20260619 14:39:59.319340 15037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:39:59.329909 15037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:39:59.332441 15037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:39:59.332592 15037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:39:59.333129 15037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:39:59.333853 15037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:39:59.333896 15037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:39:59.333945 15037 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:39:59.333972 15037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:39:59.341688 15037 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40807
I20260619 14:39:59.341758 15165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40807 every 8 connection(s)
I20260619 14:39:59.342581 15037 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:39:59.346566 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 15037
Jun 19 14:39:59 dist-test-slave-32hj krb5kdc[14126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:39:59.354971 15170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:59.347146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57372 (local address 127.0.98.62:42505)
0619 14:39:59.347997 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0619 14:39:59.348001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:39:59.348012 (+ 11us) server_negotiation.cc:407] Connection header received
0619 14:39:59.348042 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:39:59.348044 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:39:59.348084 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:39:59.348171 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:39:59.348880 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:59.349397 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:59.350284 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:59.350512 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:59.350606 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:59.351931 (+ 1325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:39:59.351935 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:39:59.351939 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:39:59.351965 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:39:59.353403 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:59.353820 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:59.353822 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:59.353824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:59.353896 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:59.354173 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:59.354176 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:59.354177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:59.354379 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:39:59.354594 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0619 14:39:59.354675 (+ 81us) server_negotiation.cc:299] Negotiation successful
0619 14:39:59.354804 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":780,"thread_start_us":72,"threads_started":1}
I20260619 14:39:59.356395 14899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:59.344216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47341 (local address 127.0.98.62:42505)
0619 14:39:59.344329 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 14:39:59.344333 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:39:59.345259 (+ 926us) server_negotiation.cc:407] Connection header received
0619 14:39:59.346487 (+ 1228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:39:59.346490 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:39:59.346546 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:39:59.346621 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:39:59.347908 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:59.348539 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:59.349749 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:59.349963 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:59.350096 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:59.352846 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:39:59.352850 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:39:59.352853 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:39:59.352892 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:39:59.354251 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:59.354798 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:59.354802 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:59.354803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:59.354886 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:39:59.355262 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:39:59.355265 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:39:59.355267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:39:59.355449 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:39:59.355528 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 14:39:59.356141 (+ 613us) server_negotiation.cc:299] Negotiation successful
0619 14:39:59.356268 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 14:39:59.357208 15168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:59.344588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42505 (local address 127.0.98.3:47341)
0619 14:39:59.345111 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:59.345151 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 14:39:59.346287 (+ 1136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:39:59.346791 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:39:59.346798 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:39:59.347167 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:39:59.347773 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:59.347789 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:59.348652 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:59.348654 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:59.349619 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:59.349629 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:59.350134 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:59.350138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:59.350301 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:59.350854 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:39:59.350874 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:39:59.352668 (+ 1794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:39:59.354356 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:39:59.354362 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:39:59.354376 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:39:59.354690 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:39:59.354975 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:39:59.354978 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:39:59.354980 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:39:59.355149 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:39:59.355580 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:39:59.355586 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:39:59.355878 (+ 292us) client_negotiation.cc:769] Sending connection context
0619 14:39:59.356061 (+ 183us) client_negotiation.cc:240] Negotiation successful
0619 14:39:59.356317 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":163,"threads_started":1}
I20260619 14:39:59.358155 15166 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42505
I20260619 14:39:59.358352 15166 heartbeater.cc:464] Registering TS with master...
I20260619 14:39:59.358822 15166 heartbeater.cc:511] Master 127.0.98.62:42505 requested a full tablet report, sending...
I20260619 14:39:59.359714 14708 ts_manager.cc:194] Registered new tserver with Master: 23b8ca310b6240ad90478bc027bf937c (127.0.98.3:40807)
I20260619 14:39:59.360204 14708 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47341
I20260619 14:39:59.368184 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:39:59.371742 14899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:59.368693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57376 (local address 127.0.98.62:42505)
0619 14:39:59.368836 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 14:39:59.368839 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:39:59.369009 (+ 170us) server_negotiation.cc:407] Connection header received
0619 14:39:59.369136 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:39:59.369138 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:39:59.369239 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:39:59.369311 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:39:59.369475 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:59.369956 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:59.370708 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:59.370900 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:59.370960 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:59.371162 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:39:59.371499 (+ 337us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:39:59.371577 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 14:39:59.371617 (+ 40us) server_negotiation.cc:299] Negotiation successful
0619 14:39:59.371654 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 14:39:59.372287 14708 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:57376:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:39:59.375305 14708 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:39:59.382551 14830 tablet_service.cc:1467] Processing CreateTablet for tablet f3973dcb94744c4ebe81bdba5358e6ef (DEFAULT_TABLE table=test-table [id=f210cf75cd8f41e1a6980ec816cfdedb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:39:59.383196 14830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3973dcb94744c4ebe81bdba5358e6ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:39:59.385790 14788 tablet_bootstrap.cc:492] T f3973dcb94744c4ebe81bdba5358e6ef P 79e520561a044d52a83a5b8258842cfc: Bootstrap starting.
I20260619 14:39:59.386612 14788 tablet_bootstrap.cc:654] T f3973dcb94744c4ebe81bdba5358e6ef P 79e520561a044d52a83a5b8258842cfc: Neither blocks nor log segments found. Creating new log.
I20260619 14:39:59.387202 15176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:59.382019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40807 (local address 127.0.0.1:58506)
0619 14:39:59.382224 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:59.382235 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:39:59.382313 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:39:59.383064 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:39:59.383067 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:39:59.383084 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:39:59.383436 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:59.383442 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:59.384950 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:59.384953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:59.385765 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:59.385775 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:59.386804 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:59.386807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:59.386896 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:59.386922 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 14:39:59.386991 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 14:39:59.387058 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":73,"spinlock_wait_cycles":768,"thread_start_us":89,"threads_started":1}
I20260619 14:39:59.387264 15177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:59.382288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58506 (local address 127.0.98.3:40807)
0619 14:39:59.382602 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0619 14:39:59.382613 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:39:59.382638 (+ 25us) server_negotiation.cc:407] Connection header received
0619 14:39:59.382711 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:39:59.382717 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:39:59.382926 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:39:59.383083 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:39:59.383668 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:59.384834 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:59.385897 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:59.386686 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:59.386786 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:59.386934 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 14:39:59.387021 (+ 87us) server_negotiation.cc:299] Negotiation successful
0619 14:39:59.387113 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":100,"threads_started":1}
I20260619 14:39:59.387480 14903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:59.381660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41403 (local address 127.0.0.1:46372)
0619 14:39:59.381778 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:59.381793 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:39:59.381865 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:39:59.383075 (+ 1210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:39:59.383078 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:39:59.383090 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:39:59.383225 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:59.383254 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:59.384668 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:59.384672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:59.385789 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:59.385799 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:59.387148 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:59.387152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:59.387240 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:59.387263 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 14:39:59.387315 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 14:39:59.387378 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":24}
I20260619 14:39:59.387826 14788 tablet_bootstrap.cc:492] T f3973dcb94744c4ebe81bdba5358e6ef P 79e520561a044d52a83a5b8258842cfc: No bootstrap required, opened a new log
I20260619 14:39:59.387883 14788 ts_tablet_manager.cc:1397] T f3973dcb94744c4ebe81bdba5358e6ef P 79e520561a044d52a83a5b8258842cfc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260619 14:39:59.388082 15175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:59.381872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46372 (local address 127.0.98.2:41403)
0619 14:39:59.382667 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0619 14:39:59.382674 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:39:59.382692 (+ 18us) server_negotiation.cc:407] Connection header received
0619 14:39:59.382759 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:39:59.382764 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:39:59.382951 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:39:59.383240 (+ 289us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:39:59.383396 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:59.384528 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:59.386150 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:59.387035 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:59.387707 (+ 672us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:59.387838 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 14:39:59.387879 (+ 41us) server_negotiation.cc:299] Negotiation successful
0619 14:39:59.387954 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":706,"thread_start_us":120,"threads_started":1}
I20260619 14:39:59.388481 14788 raft_consensus.cc:348] T f3973dcb94744c4ebe81bdba5358e6ef P 79e520561a044d52a83a5b8258842cfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5769a3dda9664950bd099d906adf485b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41403 } } peers { permanent_uuid: "23b8ca310b6240ad90478bc027bf937c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40807 } } peers { permanent_uuid: "79e520561a044d52a83a5b8258842cfc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37871 } }
I20260619 14:39:59.388600 14788 raft_consensus.cc:374] T f3973dcb94744c4ebe81bdba5358e6ef P 79e520561a044d52a83a5b8258842cfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:39:59.388644 14788 raft_consensus.cc:729] T f3973dcb94744c4ebe81bdba5358e6ef P 79e520561a044d52a83a5b8258842cfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79e520561a044d52a83a5b8258842cfc, State: Initialized, Role: FOLLOWER
I20260619 14:39:59.388783 14788 consensus_queue.cc:260] T f3973dcb94744c4ebe81bdba5358e6ef P 79e520561a044d52a83a5b8258842cfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5769a3dda9664950bd099d906adf485b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41403 } } peers { permanent_uuid: "23b8ca310b6240ad90478bc027bf937c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40807 } } peers { permanent_uuid: "79e520561a044d52a83a5b8258842cfc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37871 } }
I20260619 14:39:59.389035 14788 ts_tablet_manager.cc:1428] T f3973dcb94744c4ebe81bdba5358e6ef P 79e520561a044d52a83a5b8258842cfc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:39:59.390413 15100 tablet_service.cc:1467] Processing CreateTablet for tablet f3973dcb94744c4ebe81bdba5358e6ef (DEFAULT_TABLE table=test-table [id=f210cf75cd8f41e1a6980ec816cfdedb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:39:59.391320 15100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3973dcb94744c4ebe81bdba5358e6ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:39:59.395222 14968 tablet_service.cc:1467] Processing CreateTablet for tablet f3973dcb94744c4ebe81bdba5358e6ef (DEFAULT_TABLE table=test-table [id=f210cf75cd8f41e1a6980ec816cfdedb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:39:59.396188 14968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3973dcb94744c4ebe81bdba5358e6ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:39:59.398535 15179 tablet_bootstrap.cc:492] T f3973dcb94744c4ebe81bdba5358e6ef P 23b8ca310b6240ad90478bc027bf937c: Bootstrap starting.
I20260619 14:39:59.400326 15179 tablet_bootstrap.cc:654] T f3973dcb94744c4ebe81bdba5358e6ef P 23b8ca310b6240ad90478bc027bf937c: Neither blocks nor log segments found. Creating new log.
I20260619 14:39:59.401320 15179 log.cc:824] T f3973dcb94744c4ebe81bdba5358e6ef P 23b8ca310b6240ad90478bc027bf937c: Log is configured to *not* fsync() on all Append() calls
I20260619 14:39:59.402691 15181 tablet_bootstrap.cc:492] T f3973dcb94744c4ebe81bdba5358e6ef P 5769a3dda9664950bd099d906adf485b: Bootstrap starting.
I20260619 14:39:59.403436 15179 tablet_bootstrap.cc:492] T f3973dcb94744c4ebe81bdba5358e6ef P 23b8ca310b6240ad90478bc027bf937c: No bootstrap required, opened a new log
I20260619 14:39:59.403620 15179 ts_tablet_manager.cc:1397] T f3973dcb94744c4ebe81bdba5358e6ef P 23b8ca310b6240ad90478bc027bf937c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:39:59.404870 15181 tablet_bootstrap.cc:654] T f3973dcb94744c4ebe81bdba5358e6ef P 5769a3dda9664950bd099d906adf485b: Neither blocks nor log segments found. Creating new log.
I20260619 14:39:59.405498 15181 log.cc:824] T f3973dcb94744c4ebe81bdba5358e6ef P 5769a3dda9664950bd099d906adf485b: Log is configured to *not* fsync() on all Append() calls
I20260619 14:39:59.406394 15179 raft_consensus.cc:348] T f3973dcb94744c4ebe81bdba5358e6ef P 23b8ca310b6240ad90478bc027bf937c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5769a3dda9664950bd099d906adf485b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41403 } } peers { permanent_uuid: "23b8ca310b6240ad90478bc027bf937c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40807 } } peers { permanent_uuid: "79e520561a044d52a83a5b8258842cfc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37871 } }
I20260619 14:39:59.406597 15179 raft_consensus.cc:374] T f3973dcb94744c4ebe81bdba5358e6ef P 23b8ca310b6240ad90478bc027bf937c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:39:59.406631 15179 raft_consensus.cc:729] T f3973dcb94744c4ebe81bdba5358e6ef P 23b8ca310b6240ad90478bc027bf937c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23b8ca310b6240ad90478bc027bf937c, State: Initialized, Role: FOLLOWER
I20260619 14:39:59.407001 15181 tablet_bootstrap.cc:492] T f3973dcb94744c4ebe81bdba5358e6ef P 5769a3dda9664950bd099d906adf485b: No bootstrap required, opened a new log
I20260619 14:39:59.407016 15179 consensus_queue.cc:260] T f3973dcb94744c4ebe81bdba5358e6ef P 23b8ca310b6240ad90478bc027bf937c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5769a3dda9664950bd099d906adf485b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41403 } } peers { permanent_uuid: "23b8ca310b6240ad90478bc027bf937c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40807 } } peers { permanent_uuid: "79e520561a044d52a83a5b8258842cfc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37871 } }
I20260619 14:39:59.407166 15181 ts_tablet_manager.cc:1397] T f3973dcb94744c4ebe81bdba5358e6ef P 5769a3dda9664950bd099d906adf485b: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260619 14:39:59.407606 15166 heartbeater.cc:503] Master 127.0.98.62:42505 was elected leader, sending a full tablet report...
I20260619 14:39:59.407876 15179 ts_tablet_manager.cc:1428] T f3973dcb94744c4ebe81bdba5358e6ef P 23b8ca310b6240ad90478bc027bf937c: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260619 14:39:59.410205 15181 raft_consensus.cc:348] T f3973dcb94744c4ebe81bdba5358e6ef P 5769a3dda9664950bd099d906adf485b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5769a3dda9664950bd099d906adf485b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41403 } } peers { permanent_uuid: "23b8ca310b6240ad90478bc027bf937c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40807 } } peers { permanent_uuid: "79e520561a044d52a83a5b8258842cfc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37871 } }
I20260619 14:39:59.410487 15181 raft_consensus.cc:374] T f3973dcb94744c4ebe81bdba5358e6ef P 5769a3dda9664950bd099d906adf485b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:39:59.410553 15181 raft_consensus.cc:729] T f3973dcb94744c4ebe81bdba5358e6ef P 5769a3dda9664950bd099d906adf485b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5769a3dda9664950bd099d906adf485b, State: Initialized, Role: FOLLOWER
I20260619 14:39:59.410948 15181 consensus_queue.cc:260] T f3973dcb94744c4ebe81bdba5358e6ef P 5769a3dda9664950bd099d906adf485b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5769a3dda9664950bd099d906adf485b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41403 } } peers { permanent_uuid: "23b8ca310b6240ad90478bc027bf937c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40807 } } peers { permanent_uuid: "79e520561a044d52a83a5b8258842cfc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37871 } }
I20260619 14:39:59.411482 15034 heartbeater.cc:503] Master 127.0.98.62:42505 was elected leader, sending a full tablet report...
I20260619 14:39:59.411769 15181 ts_tablet_manager.cc:1428] T f3973dcb94744c4ebe81bdba5358e6ef P 5769a3dda9664950bd099d906adf485b: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
W20260619 14:39:59.426066 15035 tablet.cc:2369] T f3973dcb94744c4ebe81bdba5358e6ef P 5769a3dda9664950bd099d906adf485b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:39:59.466801 14902 raft_consensus.cc:482] T f3973dcb94744c4ebe81bdba5358e6ef P 79e520561a044d52a83a5b8258842cfc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:39:59.467031 14902 raft_consensus.cc:504] T f3973dcb94744c4ebe81bdba5358e6ef P 79e520561a044d52a83a5b8258842cfc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5769a3dda9664950bd099d906adf485b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41403 } } peers { permanent_uuid: "23b8ca310b6240ad90478bc027bf937c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40807 } } peers { permanent_uuid: "79e520561a044d52a83a5b8258842cfc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37871 } }
I20260619 14:39:59.468111 14902 leader_election.cc:288] T f3973dcb94744c4ebe81bdba5358e6ef P 79e520561a044d52a83a5b8258842cfc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5769a3dda9664950bd099d906adf485b (127.0.98.2:41403), 23b8ca310b6240ad90478bc027bf937c (127.0.98.3:40807)
I20260619 14:39:59.471737 15177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:59.468545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35295 (local address 127.0.98.3:40807)
0619 14:39:59.468658 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 14:39:59.468662 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:39:59.468850 (+ 188us) server_negotiation.cc:407] Connection header received
0619 14:39:59.468943 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:39:59.468946 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:39:59.468997 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:39:59.469072 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:39:59.469354 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:59.469842 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:59.470832 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:59.471244 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:59.471338 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:59.471397 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 14:39:59.471571 (+ 174us) server_negotiation.cc:299] Negotiation successful
0619 14:39:59.471639 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 14:39:59.471737 14898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:59.468351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41403 (local address 127.0.98.1:58407)
0619 14:39:59.468500 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:59.468514 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:39:59.468601 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:39:59.468879 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:39:59.468882 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:39:59.468902 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:39:59.469026 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:59.469033 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:59.469843 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:59.469847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:59.470652 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:59.470660 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:59.471326 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:59.471330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:59.471423 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:59.471445 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 14:39:59.471504 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 14:39:59.471571 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260619 14:39:59.471735 15175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:59.468455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58407 (local address 127.0.98.2:41403)
0619 14:39:59.468574 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 14:39:59.468578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:39:59.468595 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:39:59.468695 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:39:59.468699 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:39:59.468775 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:39:59.468851 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:39:59.469150 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:59.469726 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:59.470779 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:59.471196 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:59.471295 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:59.471362 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 14:39:59.471539 (+ 177us) server_negotiation.cc:299] Negotiation successful
0619 14:39:59.471609 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 14:39:59.471765 15185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:59.468379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40807 (local address 127.0.98.1:35295)
0619 14:39:59.468763 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:59.468775 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:39:59.468866 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:39:59.469118 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:39:59.469121 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:39:59.469141 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:39:59.469254 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:59.469261 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:59.469939 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:59.469941 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:59.470688 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:39:59.470696 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:39:59.471380 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:39:59.471384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:39:59.471459 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:59.471475 (+ 16us) client_negotiation.cc:769] Sending connection context
0619 14:39:59.471545 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 14:39:59.471609 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":82,"threads_started":1}
I20260619 14:39:59.472330 14988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3973dcb94744c4ebe81bdba5358e6ef" candidate_uuid: "79e520561a044d52a83a5b8258842cfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5769a3dda9664950bd099d906adf485b" is_pre_election: true
I20260619 14:39:59.472332 15120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3973dcb94744c4ebe81bdba5358e6ef" candidate_uuid: "79e520561a044d52a83a5b8258842cfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b8ca310b6240ad90478bc027bf937c" is_pre_election: true
I20260619 14:39:59.472635 15120 raft_consensus.cc:2454] T f3973dcb94744c4ebe81bdba5358e6ef P 23b8ca310b6240ad90478bc027bf937c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79e520561a044d52a83a5b8258842cfc in term 0.
I20260619 14:39:59.472635 14988 raft_consensus.cc:2454] T f3973dcb94744c4ebe81bdba5358e6ef P 5769a3dda9664950bd099d906adf485b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79e520561a044d52a83a5b8258842cfc in term 0.
I20260619 14:39:59.473129 14782 leader_election.cc:302] T f3973dcb94744c4ebe81bdba5358e6ef P 79e520561a044d52a83a5b8258842cfc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 23b8ca310b6240ad90478bc027bf937c, 5769a3dda9664950bd099d906adf485b, 79e520561a044d52a83a5b8258842cfc; no voters:
I20260619 14:39:59.473373 14902 raft_consensus.cc:2781] T f3973dcb94744c4ebe81bdba5358e6ef P 79e520561a044d52a83a5b8258842cfc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:39:59.473448 14902 raft_consensus.cc:482] T f3973dcb94744c4ebe81bdba5358e6ef P 79e520561a044d52a83a5b8258842cfc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:39:59.473491 14902 raft_consensus.cc:3037] T f3973dcb94744c4ebe81bdba5358e6ef P 79e520561a044d52a83a5b8258842cfc [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:39:59.474478 14902 raft_consensus.cc:504] T f3973dcb94744c4ebe81bdba5358e6ef P 79e520561a044d52a83a5b8258842cfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5769a3dda9664950bd099d906adf485b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41403 } } peers { permanent_uuid: "23b8ca310b6240ad90478bc027bf937c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40807 } } peers { permanent_uuid: "79e520561a044d52a83a5b8258842cfc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37871 } }
I20260619 14:39:59.474808 14902 leader_election.cc:288] T f3973dcb94744c4ebe81bdba5358e6ef P 79e520561a044d52a83a5b8258842cfc [CANDIDATE]: Term 1 election: Requested vote from peers 5769a3dda9664950bd099d906adf485b (127.0.98.2:41403), 23b8ca310b6240ad90478bc027bf937c (127.0.98.3:40807)
I20260619 14:39:59.475148 15120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3973dcb94744c4ebe81bdba5358e6ef" candidate_uuid: "79e520561a044d52a83a5b8258842cfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b8ca310b6240ad90478bc027bf937c"
I20260619 14:39:59.475180 14988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3973dcb94744c4ebe81bdba5358e6ef" candidate_uuid: "79e520561a044d52a83a5b8258842cfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5769a3dda9664950bd099d906adf485b"
I20260619 14:39:59.475279 14988 raft_consensus.cc:3037] T f3973dcb94744c4ebe81bdba5358e6ef P 5769a3dda9664950bd099d906adf485b [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:39:59.475281 15120 raft_consensus.cc:3037] T f3973dcb94744c4ebe81bdba5358e6ef P 23b8ca310b6240ad90478bc027bf937c [term 0 FOLLOWER]: Advancing to term 1
W20260619 14:39:59.475692 14897 tablet.cc:2369] T f3973dcb94744c4ebe81bdba5358e6ef P 79e520561a044d52a83a5b8258842cfc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:39:59.476490 15120 raft_consensus.cc:2454] T f3973dcb94744c4ebe81bdba5358e6ef P 23b8ca310b6240ad90478bc027bf937c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79e520561a044d52a83a5b8258842cfc in term 1.
I20260619 14:39:59.476502 14988 raft_consensus.cc:2454] T f3973dcb94744c4ebe81bdba5358e6ef P 5769a3dda9664950bd099d906adf485b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79e520561a044d52a83a5b8258842cfc in term 1.
I20260619 14:39:59.476790 14784 leader_election.cc:302] T f3973dcb94744c4ebe81bdba5358e6ef P 79e520561a044d52a83a5b8258842cfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 23b8ca310b6240ad90478bc027bf937c, 5769a3dda9664950bd099d906adf485b, 79e520561a044d52a83a5b8258842cfc; no voters:
I20260619 14:39:59.476953 14902 raft_consensus.cc:2781] T f3973dcb94744c4ebe81bdba5358e6ef P 79e520561a044d52a83a5b8258842cfc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:39:59.477166 14902 raft_consensus.cc:686] T f3973dcb94744c4ebe81bdba5358e6ef P 79e520561a044d52a83a5b8258842cfc [term 1 LEADER]: Becoming Leader. State: Replica: 79e520561a044d52a83a5b8258842cfc, State: Running, Role: LEADER
I20260619 14:39:59.477411 14902 consensus_queue.cc:237] T f3973dcb94744c4ebe81bdba5358e6ef P 79e520561a044d52a83a5b8258842cfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5769a3dda9664950bd099d906adf485b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41403 } } peers { permanent_uuid: "23b8ca310b6240ad90478bc027bf937c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40807 } } peers { permanent_uuid: "79e520561a044d52a83a5b8258842cfc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37871 } }
I20260619 14:39:59.480050 14706 catalog_manager.cc:5495] T f3973dcb94744c4ebe81bdba5358e6ef P 79e520561a044d52a83a5b8258842cfc reported cstate change: term changed from 0 to 1, leader changed from <none> to 79e520561a044d52a83a5b8258842cfc (127.0.98.1). New cstate: current_term: 1 leader_uuid: "79e520561a044d52a83a5b8258842cfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5769a3dda9664950bd099d906adf485b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23b8ca310b6240ad90478bc027bf937c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79e520561a044d52a83a5b8258842cfc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37871 } health_report { overall_health: HEALTHY } } }
I20260619 14:39:59.507148 14904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:59.504141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57396 (local address 127.0.98.1:37871)
0619 14:39:59.504251 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 14:39:59.504255 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:39:59.504268 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:39:59.504363 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:39:59.504365 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:39:59.504418 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:39:59.504503 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:39:59.504747 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:59.505246 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:59.506007 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:39:59.506220 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:39:59.506295 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:39:59.506471 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:39:59.506875 (+ 404us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:39:59.506955 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 14:39:59.506995 (+ 40us) server_negotiation.cc:299] Negotiation successful
0619 14:39:59.507043 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 14:39:59.510769 14988 raft_consensus.cc:1261] T f3973dcb94744c4ebe81bdba5358e6ef P 5769a3dda9664950bd099d906adf485b [term 1 FOLLOWER]: Refusing update from remote peer 79e520561a044d52a83a5b8258842cfc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:39:59.510769 15120 raft_consensus.cc:1261] T f3973dcb94744c4ebe81bdba5358e6ef P 23b8ca310b6240ad90478bc027bf937c [term 1 FOLLOWER]: Refusing update from remote peer 79e520561a044d52a83a5b8258842cfc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:39:59.511495 15186 consensus_queue.cc:1035] T f3973dcb94744c4ebe81bdba5358e6ef P 79e520561a044d52a83a5b8258842cfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5769a3dda9664950bd099d906adf485b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:39:59.511725 14902 consensus_queue.cc:1035] T f3973dcb94744c4ebe81bdba5358e6ef P 79e520561a044d52a83a5b8258842cfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "23b8ca310b6240ad90478bc027bf937c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:39:59.525986 14706 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:57376:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:39:59.526163 14706 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:57376:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:39:59.528549 14706 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711: Sending DeleteTablet for 3 replicas of tablet f3973dcb94744c4ebe81bdba5358e6ef
I20260619 14:39:59.529321 14968 tablet_service.cc:1514] Processing DeleteTablet for tablet f3973dcb94744c4ebe81bdba5358e6ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:39:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46372
I20260619 14:39:59.529322 15100 tablet_service.cc:1514] Processing DeleteTablet for tablet f3973dcb94744c4ebe81bdba5358e6ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:39:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58506
I20260619 14:39:59.529544 14830 tablet_service.cc:1514] Processing DeleteTablet for tablet f3973dcb94744c4ebe81bdba5358e6ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:39:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57390
I20260619 14:39:59.529912 15201 tablet_replica.cc:330] T f3973dcb94744c4ebe81bdba5358e6ef P 23b8ca310b6240ad90478bc027bf937c: stopping tablet replica
I20260619 14:39:59.530108 15202 tablet_replica.cc:330] T f3973dcb94744c4ebe81bdba5358e6ef P 79e520561a044d52a83a5b8258842cfc: stopping tablet replica
I20260619 14:39:59.530103 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 14746
I20260619 14:39:59.530660 15201 raft_consensus.cc:2229] T f3973dcb94744c4ebe81bdba5358e6ef P 23b8ca310b6240ad90478bc027bf937c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:39:59.530961 15201 raft_consensus.cc:2258] T f3973dcb94744c4ebe81bdba5358e6ef P 23b8ca310b6240ad90478bc027bf937c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:39:59.531286 15200 tablet_replica.cc:330] T f3973dcb94744c4ebe81bdba5358e6ef P 5769a3dda9664950bd099d906adf485b: stopping tablet replica
I20260619 14:39:59.531533 15200 raft_consensus.cc:2229] T f3973dcb94744c4ebe81bdba5358e6ef P 5769a3dda9664950bd099d906adf485b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:39:59.531774 15200 raft_consensus.cc:2258] T f3973dcb94744c4ebe81bdba5358e6ef P 5769a3dda9664950bd099d906adf485b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:39:59.532613 15201 ts_tablet_manager.cc:1905] T f3973dcb94744c4ebe81bdba5358e6ef P 23b8ca310b6240ad90478bc027bf937c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:39:59.532923 15200 ts_tablet_manager.cc:1905] T f3973dcb94744c4ebe81bdba5358e6ef P 5769a3dda9664950bd099d906adf485b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:39:59.534998 15200 ts_tablet_manager.cc:1918] T f3973dcb94744c4ebe81bdba5358e6ef P 5769a3dda9664950bd099d906adf485b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:39:59.535094 15200 log.cc:1196] T f3973dcb94744c4ebe81bdba5358e6ef P 5769a3dda9664950bd099d906adf485b: Deleting WAL directory at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-1/wal/wals/f3973dcb94744c4ebe81bdba5358e6ef
I20260619 14:39:59.535228 15201 ts_tablet_manager.cc:1918] T f3973dcb94744c4ebe81bdba5358e6ef P 23b8ca310b6240ad90478bc027bf937c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:39:59.535332 15201 log.cc:1196] T f3973dcb94744c4ebe81bdba5358e6ef P 23b8ca310b6240ad90478bc027bf937c: Deleting WAL directory at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-2/wal/wals/f3973dcb94744c4ebe81bdba5358e6ef
I20260619 14:39:59.535380 15200 ts_tablet_manager.cc:1939] T f3973dcb94744c4ebe81bdba5358e6ef P 5769a3dda9664950bd099d906adf485b: Deleting consensus metadata
I20260619 14:39:59.535656 15201 ts_tablet_manager.cc:1939] T f3973dcb94744c4ebe81bdba5358e6ef P 23b8ca310b6240ad90478bc027bf937c: Deleting consensus metadata
I20260619 14:39:59.536486 14694 catalog_manager.cc:4841] TS 5769a3dda9664950bd099d906adf485b (127.0.98.2:41403): tablet f3973dcb94744c4ebe81bdba5358e6ef (table test-table [id=f210cf75cd8f41e1a6980ec816cfdedb]) successfully deleted
I20260619 14:39:59.536759 14692 catalog_manager.cc:4841] TS 23b8ca310b6240ad90478bc027bf937c (127.0.98.3:40807): tablet f3973dcb94744c4ebe81bdba5358e6ef (table test-table [id=f210cf75cd8f41e1a6980ec816cfdedb]) successfully deleted
W20260619 14:39:59.537349 14693 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37871 (error 108)
I20260619 14:39:59.537554 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 14900
I20260619 14:39:59.538314 14903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:59.538061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37871 (local address 127.0.0.1:57406)
0619 14:39:59.538165 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:59.538234 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260619 14:39:59.542343 14693 catalog_manager.cc:4584] TS 79e520561a044d52a83a5b8258842cfc (127.0.98.1:37871): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f3973dcb94744c4ebe81bdba5358e6ef: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37871: connect: Connection refused (error 111)
I20260619 14:39:59.542966 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 15037
I20260619 14:39:59.547362 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 14675
I20260619 14:39:59.552285 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42505
--webserver_interface=127.0.98.62
--webserver_port=34235
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42505
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:39:59.656350 15204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:39:59.656585 15204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:39:59.656622 15204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:39:59.660513 15204 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:39:59.660562 15204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:39:59.660579 15204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:39:59.660594 15204 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:39:59.660612 15204 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:39:59.665582 15204 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42505
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=34235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:39:59.666463 15204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:39:59.667358 15204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:39:59.674990 15212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:39:59.675014 15209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:39:59.675034 15210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:39:59.675138 15204 server_base.cc:1034] running on GCE node
I20260619 14:39:59.675674 15204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:39:59.676633 15204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:39:59.677780 15204 hybrid_clock.cc:648] HybridClock initialized: now 1781879999677756 us; error 36 us; skew 500 ppm
Jun 19 14:39:59 dist-test-slave-32hj krb5kdc[14126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:39:59.680233 15204 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:39:59.681058 15204 webserver.cc:460] Webserver started at http://127.0.98.62:34235/ using document root <none> and password file <none>
I20260619 14:39:59.681540 15204 fs_manager.cc:359] Metadata directory not provided
I20260619 14:39:59.681600 15204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:39:59.684976 15204 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:39:59.686839 15219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:39:59.687906 15204 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:39:59.688015 15204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/master-0/wal
uuid: "2cd8db0aede74336bcb242117ad8d711"
format_stamp: "Formatted at 2026-06-19 14:39:57 on dist-test-slave-32hj"
server_key: "f841e6a7a022fdbbefa013b7c11ded5c"
server_key_iv: "56ae045aede0fd73e4fe6749583233f8"
server_key_version: "encryptionkey@0"
I20260619 14:39:59.688500 15204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:39:59.712832 15204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:39:59.715451 15204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:39:59.715610 15204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:39:59.722244 15204 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42505
I20260619 14:39:59.722263 15271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42505 every 8 connection(s)
I20260619 14:39:59.723192 15204 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:39:59.727216 15272 sys_catalog.cc:263] Verifying existing consensus state
I20260619 14:39:59.727766 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 15204
I20260619 14:39:59.728140 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37871
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:42505
--webserver_port=39381
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:39:59.728892 15272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711: Bootstrap starting.
I20260619 14:39:59.745343 15272 log.cc:824] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711: Log is configured to *not* fsync() on all Append() calls
I20260619 14:39:59.756281 15272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:39:59.756595 15272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711: Bootstrap complete.
I20260619 14:39:59.760923 15272 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cd8db0aede74336bcb242117ad8d711" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42505 } }
I20260619 14:39:59.761317 15272 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cd8db0aede74336bcb242117ad8d711, State: Initialized, Role: FOLLOWER
I20260619 14:39:59.761780 15272 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cd8db0aede74336bcb242117ad8d711" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42505 } }
I20260619 14:39:59.761900 15272 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:39:59.761963 15272 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:39:59.762055 15272 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711 [term 2 FOLLOWER]: Advancing to term 3
I20260619 14:39:59.763801 15272 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cd8db0aede74336bcb242117ad8d711" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42505 } }
I20260619 14:39:59.764098 15272 leader_election.cc:302] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2cd8db0aede74336bcb242117ad8d711; no voters:
I20260619 14:39:59.764369 15272 leader_election.cc:288] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 14:39:59.764510 15276 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 14:39:59.764746 15276 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711 [term 3 LEADER]: Becoming Leader. State: Replica: 2cd8db0aede74336bcb242117ad8d711, State: Running, Role: LEADER
I20260619 14:39:59.765087 15276 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cd8db0aede74336bcb242117ad8d711" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42505 } }
I20260619 14:39:59.765545 15272 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:39:59.765956 15276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2cd8db0aede74336bcb242117ad8d711. Latest consensus state: current_term: 3 leader_uuid: "2cd8db0aede74336bcb242117ad8d711" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cd8db0aede74336bcb242117ad8d711" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42505 } } }
I20260619 14:39:59.766072 15276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711 [sys.catalog]: This master's current role is: LEADER
I20260619 14:39:59.766402 15277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "2cd8db0aede74336bcb242117ad8d711" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cd8db0aede74336bcb242117ad8d711" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42505 } } }
I20260619 14:39:59.766561 15277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711 [sys.catalog]: This master's current role is: LEADER
I20260619 14:39:59.770740 15291 catalog_manager.cc:1259] Loaded cluster ID: bc4cc76e8b3a4e5b997014d7858a35b3
I20260619 14:39:59.770792 15291 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711: loading cluster ID for follower catalog manager: success
W20260619 14:39:59.771760 15291 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 2cd8db0aede74336bcb242117ad8d711: acquiring CA information for follower catalog manager: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
W20260619 14:39:59.771824 15291 catalog_manager.cc:873] Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO: failed to prepare follower catalog manager, will retry
I20260619 14:39:59.771869 15280 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:39:59.774667 15280 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:39:59.775146 15280 catalog_manager.cc:1259] Loaded cluster ID: bc4cc76e8b3a4e5b997014d7858a35b3
I20260619 14:39:59.775200 15280 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260619 14:39:59.775800 15280 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fae7b2789ad google::LogMessage::Fail() at ??:0
@ 0x7fae7b27b88a google::LogMessage::SendToLog() at ??:0
@ 0x7fae7b2784de google::LogMessage::Flush() at ??:0
@ 0x7fae7b279a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fae85bb7fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fae85bb8d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fae85bb5579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fae85bf9640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fae7d4ca3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fae7d4cac83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fae7d4cc3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fae7d4bb512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fae7eaff6db start_thread at ??:0
@ 0x7fae79d7661f clone at ??:0
W20260619 14:39:59.859769 15274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:39:59.860030 15274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:39:59.860080 15274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:39:59.864018 15274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:39:59.864065 15274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:39:59.864123 15274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:39:59.869109 15274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37871
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=39381
--tserver_master_addrs=127.0.98.62:42505
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:39:59.870028 15274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:39:59.870954 15274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:39:59.879592 15314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:39:59.879608 15313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:39:59.879688 15274 server_base.cc:1034] running on GCE node
W20260619 14:39:59.879817 15316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:39:59.880285 15274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:39:59.881018 15274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:39:59.882217 15274 hybrid_clock.cc:648] HybridClock initialized: now 1781879999882177 us; error 47 us; skew 500 ppm
Jun 19 14:39:59 dist-test-slave-32hj krb5kdc[14126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:39:59.885360 15274 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:39:59.886353 15274 webserver.cc:460] Webserver started at http://127.0.98.1:39381/ using document root <none> and password file <none>
I20260619 14:39:59.887003 15274 fs_manager.cc:359] Metadata directory not provided
I20260619 14:39:59.887092 15274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:39:59.891780 15274 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:39:59.894224 15323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:39:59.895571 15274 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:39:59.895682 15274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-0/wal
uuid: "79e520561a044d52a83a5b8258842cfc"
format_stamp: "Formatted at 2026-06-19 14:39:57 on dist-test-slave-32hj"
server_key: "7b52a60cbb6be4cac98600c04047f7b6"
server_key_iv: "5e58e9f862ec1996b622c6ed2971d876"
server_key_version: "encryptionkey@0"
I20260619 14:39:59.896341 15274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:39:59.926884 15274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:39:59.930678 15274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:39:59.930871 15274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:39:59.931574 15274 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:39:59.932688 15330 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 14:39:59.936545 15274 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 14:39:59.936619 15274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260619 14:39:59.936690 15274 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 14:39:59.938759 15274 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 14:39:59.938814 15274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260619 14:39:59.939108 15330 tablet_bootstrap.cc:492] T f3973dcb94744c4ebe81bdba5358e6ef P 79e520561a044d52a83a5b8258842cfc: Bootstrap starting.
I20260619 14:39:59.949157 15274 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37871
I20260619 14:39:59.949472 15437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37871 every 8 connection(s)
I20260619 14:39:59.950359 15274 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:39:59.954249 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 15274
I20260619 14:39:59.954671 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:41403
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:42505
--webserver_port=41345
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:39:59.958340 15330 log.cc:824] T f3973dcb94744c4ebe81bdba5358e6ef P 79e520561a044d52a83a5b8258842cfc: Log is configured to *not* fsync() on all Append() calls
I20260619 14:39:59.967243 15330 tablet_bootstrap.cc:492] T f3973dcb94744c4ebe81bdba5358e6ef P 79e520561a044d52a83a5b8258842cfc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:39:59.967607 15330 tablet_bootstrap.cc:492] T f3973dcb94744c4ebe81bdba5358e6ef P 79e520561a044d52a83a5b8258842cfc: Bootstrap complete.
I20260619 14:39:59.968900 15330 ts_tablet_manager.cc:1397] T f3973dcb94744c4ebe81bdba5358e6ef P 79e520561a044d52a83a5b8258842cfc: Time spent bootstrapping tablet: real 0.030s user 0.025s sys 0.004s
I20260619 14:39:59.971949 15330 raft_consensus.cc:348] T f3973dcb94744c4ebe81bdba5358e6ef P 79e520561a044d52a83a5b8258842cfc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5769a3dda9664950bd099d906adf485b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41403 } } peers { permanent_uuid: "23b8ca310b6240ad90478bc027bf937c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40807 } } peers { permanent_uuid: "79e520561a044d52a83a5b8258842cfc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37871 } }
I20260619 14:39:59.972491 15330 raft_consensus.cc:729] T f3973dcb94744c4ebe81bdba5358e6ef P 79e520561a044d52a83a5b8258842cfc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79e520561a044d52a83a5b8258842cfc, State: Initialized, Role: FOLLOWER
I20260619 14:39:59.973079 15330 consensus_queue.cc:260] T f3973dcb94744c4ebe81bdba5358e6ef P 79e520561a044d52a83a5b8258842cfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5769a3dda9664950bd099d906adf485b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41403 } } peers { permanent_uuid: "23b8ca310b6240ad90478bc027bf937c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40807 } } peers { permanent_uuid: "79e520561a044d52a83a5b8258842cfc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37871 } }
I20260619 14:39:59.974115 15330 ts_tablet_manager.cc:1428] T f3973dcb94744c4ebe81bdba5358e6ef P 79e520561a044d52a83a5b8258842cfc: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260619 14:39:59.987919 15440 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:39:59.952569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42505 (local address 127.0.98.1:41885)
0619 14:39:59.953146 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:59.953222 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0619 14:39:59.954429 (+ 1207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:39:59.987060 (+ 32631us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42505: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":178,"threads_started":1}
W20260619 14:39:59.988456 15438 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42505 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:42505: Client connection negotiation failed: client connection to 127.0.98.62:42505: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 14:39:59.989151 15440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:59.988899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42505 (local address 127.0.98.1:52571)
0619 14:39:59.989021 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:59.989089 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260619 14:39:59.989771 15440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:39:59.989549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42505 (local address 127.0.98.1:44991)
0619 14:39:59.989648 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0619 14:39:59.989697 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260619 14:39:59.989939 15438 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 14:40:00.067329 15441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:40:00.067571 15441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:40:00.067611 15441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:40:00.071462 15441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:40:00.071509 15441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:40:00.071563 15441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:40:00.076534 15441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:41403
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=41345
--tserver_master_addrs=127.0.98.62:42505
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:40:00.077481 15441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:40:00.078388 15441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:40:00.087325 15448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:40:00.087333 15449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:40:00.087333 15451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:40:00.087484 15441 server_base.cc:1034] running on GCE node
I20260619 14:40:00.087949 15441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:40:00.088510 15441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:40:00.089659 15441 hybrid_clock.cc:648] HybridClock initialized: now 1781880000089621 us; error 48 us; skew 500 ppm
Jun 19 14:40:00 dist-test-slave-32hj krb5kdc[14126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880000, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:40:00.092121 15441 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:40:00.092931 15441 webserver.cc:460] Webserver started at http://127.0.98.2:41345/ using document root <none> and password file <none>
I20260619 14:40:00.093425 15441 fs_manager.cc:359] Metadata directory not provided
I20260619 14:40:00.093487 15441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:40:00.097075 15441 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260619 14:40:00.098960 15458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:40:00.100078 15441 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:40:00.100212 15441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-1/wal
uuid: "5769a3dda9664950bd099d906adf485b"
format_stamp: "Formatted at 2026-06-19 14:39:57 on dist-test-slave-32hj"
server_key: "4d8343bc9b01d6896b01a032c9168c6e"
server_key_iv: "1fedf320a17bf400182d154a57caf0d8"
server_key_version: "encryptionkey@0"
I20260619 14:40:00.100750 15441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:40:00.114625 15441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:40:00.117239 15441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:40:00.117403 15441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:40:00.117944 15441 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:40:00.118703 15441 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:40:00.118749 15441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:40:00.118798 15441 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:40:00.118824 15441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:40:00.128362 15441 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41403
I20260619 14:40:00.128428 15571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41403 every 8 connection(s)
I20260619 14:40:00.129549 15441 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:40:00.132225 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 15441
I20260619 14:40:00.132639 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:40807
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:42505
--webserver_port=39511
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:40:00.133739 15574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:00.131871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42505 (local address 127.0.98.2:48229)
0619 14:40:00.132383 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:00.132514 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":114,"threads_started":1}
W20260619 14:40:00.134135 15572 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42505 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:42505: Client connection negotiation failed: client connection to 127.0.98.62:42505: connect: Connection refused (error 111)
I20260619 14:40:00.134729 15574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:00.134484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42505 (local address 127.0.98.2:42079)
0619 14:40:00.134600 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:00.134654 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260619 14:40:00.135308 15574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:00.135107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42505 (local address 127.0.98.2:44753)
0619 14:40:00.135207 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:00.135253 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260619 14:40:00.135514 15572 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 14:40:00.236395 15575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:40:00.236618 15575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:40:00.236656 15575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:40:00.240504 15575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:40:00.240552 15575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:40:00.240608 15575 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:40:00.245530 15575 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:40807
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=39511
--tserver_master_addrs=127.0.98.62:42505
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:40:00.246423 15575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:40:00.247334 15575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:40:00.256104 15583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:40:00.256142 15581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:40:00.256155 15575 server_base.cc:1034] running on GCE node
W20260619 14:40:00.256103 15580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:40:00.256752 15575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:40:00.257345 15575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:40:00.258494 15575 hybrid_clock.cc:648] HybridClock initialized: now 1781880000258483 us; error 35 us; skew 500 ppm
Jun 19 14:40:00 dist-test-slave-32hj krb5kdc[14126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880000, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:40:00.261088 15575 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:40:00.261916 15575 webserver.cc:460] Webserver started at http://127.0.98.3:39511/ using document root <none> and password file <none>
I20260619 14:40:00.262426 15575 fs_manager.cc:359] Metadata directory not provided
I20260619 14:40:00.262490 15575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:40:00.265954 15575 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:40:00.267841 15590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:40:00.269032 15575 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:40:00.269129 15575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-2/wal
uuid: "23b8ca310b6240ad90478bc027bf937c"
format_stamp: "Formatted at 2026-06-19 14:39:58 on dist-test-slave-32hj"
server_key: "9ffa6d0e2ca813649ace55467416ec14"
server_key_iv: "60996965574646544344e5d5d02915a1"
server_key_version: "encryptionkey@0"
I20260619 14:40:00.269621 15575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:40:00.284469 15575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:40:00.286886 15575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:40:00.287031 15575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:40:00.287534 15575 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:40:00.288247 15575 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:40:00.288292 15575 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:40:00.288339 15575 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:40:00.288365 15575 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:40:00.296610 15575 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40807
I20260619 14:40:00.296638 15703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40807 every 8 connection(s)
I20260619 14:40:00.297503 15575 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:40:00.298493 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 15575
I20260619 14:40:00.300990 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:00.299364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42505 (local address 127.0.98.3:37285)
0619 14:40:00.299778 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:00.299872 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":112,"threads_started":1}
W20260619 14:40:00.301409 15704 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42505 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:42505: Client connection negotiation failed: client connection to 127.0.98.62:42505: connect: Connection refused (error 111)
I20260619 14:40:00.301924 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:00.301724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42505 (local address 127.0.98.3:38503)
0619 14:40:00.301832 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:00.301875 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 14:40:00.302449 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:00.302278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42505 (local address 127.0.98.3:43671)
0619 14:40:00.302360 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:00.302398 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260619 14:40:00.302595 15704 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 14:40:00.990906 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:00.990497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42505 (local address 127.0.98.1:48511)
0619 14:40:00.990742 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:00.990797 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":75,"threads_started":1}
I20260619 14:40:01.136507 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:01.136070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42505 (local address 127.0.98.2:53851)
0619 14:40:01.136345 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:01.136405 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":69,"threads_started":1}
I20260619 14:40:01.303447 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:01.303084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42505 (local address 127.0.98.3:57093)
0619 14:40:01.303287 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:01.303340 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":64,"threads_started":1}
I20260619 14:40:01.783895 15711 raft_consensus.cc:482] T f3973dcb94744c4ebe81bdba5358e6ef P 79e520561a044d52a83a5b8258842cfc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:40:01.784065 15711 raft_consensus.cc:504] T f3973dcb94744c4ebe81bdba5358e6ef P 79e520561a044d52a83a5b8258842cfc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5769a3dda9664950bd099d906adf485b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41403 } } peers { permanent_uuid: "23b8ca310b6240ad90478bc027bf937c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40807 } } peers { permanent_uuid: "79e520561a044d52a83a5b8258842cfc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37871 } }
I20260619 14:40:01.785915 15711 leader_election.cc:288] T f3973dcb94744c4ebe81bdba5358e6ef P 79e520561a044d52a83a5b8258842cfc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5769a3dda9664950bd099d906adf485b (127.0.98.2:41403), 23b8ca310b6240ad90478bc027bf937c (127.0.98.3:40807)
Jun 19 14:40:01 dist-test-slave-32hj krb5kdc[14126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Jun 19 14:40:01 dist-test-slave-32hj krb5kdc[14126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781879999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260619 14:40:01.800927 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:01.785972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51295 (local address 127.0.98.2:41403)
0619 14:40:01.786254 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0619 14:40:01.786267 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:40:01.786305 (+ 38us) server_negotiation.cc:407] Connection header received
0619 14:40:01.787009 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:40:01.787016 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:40:01.787373 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:40:01.787650 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:40:01.789134 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:40:01.790661 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:40:01.792356 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:40:01.793120 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:40:01.793276 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:40:01.796383 (+ 3107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:40:01.796390 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:40:01.796407 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:40:01.796442 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:40:01.798443 (+ 2001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:40:01.798893 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:40:01.798898 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:40:01.798904 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:40:01.799045 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:40:01.799301 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:40:01.799304 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:40:01.799306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:40:01.799703 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:40:01.799824 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 14:40:01.800370 (+ 546us) server_negotiation.cc:299] Negotiation successful
0619 14:40:01.800615 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":97,"threads_started":1}
I20260619 14:40:01.801450 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:01.786329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55281 (local address 127.0.98.3:40807)
0619 14:40:01.786578 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0619 14:40:01.786600 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 14:40:01.786762 (+ 162us) server_negotiation.cc:407] Connection header received
0619 14:40:01.787450 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:40:01.787456 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:40:01.787756 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:40:01.787997 (+ 241us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:40:01.789124 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:40:01.790848 (+ 1724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:40:01.792370 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:40:01.793086 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:40:01.793238 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:40:01.796057 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:40:01.796063 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:40:01.796081 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:40:01.796114 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:40:01.798217 (+ 2103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:40:01.798721 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:40:01.798727 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:40:01.798733 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:40:01.798886 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:40:01.799261 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:40:01.799265 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:40:01.799266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:40:01.799790 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:40:01.799917 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 14:40:01.800571 (+ 654us) server_negotiation.cc:299] Negotiation successful
0619 14:40:01.800908 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":62,"threads_started":1}
I20260619 14:40:01.801625 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:01.785697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41403 (local address 127.0.98.1:51295)
0619 14:40:01.785997 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:01.786095 (+ 98us) client_negotiation.cc:174] Beginning negotiation
0619 14:40:01.786211 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:40:01.787732 (+ 1521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:40:01.787738 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:40:01.788244 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:40:01.788938 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:40:01.788959 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:40:01.790841 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:40:01.790845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:40:01.792241 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:40:01.792252 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:40:01.793449 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:40:01.793453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:40:01.793552 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:40:01.794083 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:40:01.794097 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:40:01.796164 (+ 2067us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:40:01.798555 (+ 2391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:40:01.798558 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:40:01.798560 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:40:01.798776 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:40:01.799147 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:40:01.799150 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:40:01.799152 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:40:01.799230 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:40:01.799813 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:40:01.799819 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:40:01.800065 (+ 246us) client_negotiation.cc:769] Sending connection context
0619 14:40:01.800260 (+ 195us) client_negotiation.cc:240] Negotiation successful
0619 14:40:01.800577 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":129,"spinlock_wait_cycles":28928,"thread_start_us":69,"threads_started":1}
I20260619 14:40:01.802078 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:01.785710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40807 (local address 127.0.98.1:55281)
0619 14:40:01.786657 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:01.786678 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 14:40:01.786773 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:40:01.787962 (+ 1189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:40:01.787965 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:40:01.788242 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:40:01.788937 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:40:01.788958 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:40:01.791062 (+ 2104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:40:01.791065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:40:01.792243 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:40:01.792252 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:40:01.793199 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:40:01.793203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:40:01.793372 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:40:01.793961 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:40:01.793983 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:40:01.795797 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:40:01.798327 (+ 2530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:40:01.798333 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:40:01.798346 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:40:01.798615 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:40:01.798978 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:40:01.798981 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:40:01.798983 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:40:01.799168 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:40:01.799966 (+ 798us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:40:01.799969 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:40:01.800072 (+ 103us) client_negotiation.cc:769] Sending connection context
0619 14:40:01.800260 (+ 188us) client_negotiation.cc:240] Negotiation successful
0619 14:40:01.801809 (+ 1549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":888,"mutex_wait_us":543,"thread_start_us":64,"threads_started":1}
W20260619 14:40:01.804513 15326 leader_election.cc:341] T f3973dcb94744c4ebe81bdba5358e6ef P 79e520561a044d52a83a5b8258842cfc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5769a3dda9664950bd099d906adf485b (127.0.98.2:41403): Not found: Tablet not found: f3973dcb94744c4ebe81bdba5358e6ef
W20260619 14:40:01.805200 15324 leader_election.cc:341] T f3973dcb94744c4ebe81bdba5358e6ef P 79e520561a044d52a83a5b8258842cfc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 23b8ca310b6240ad90478bc027bf937c (127.0.98.3:40807): Not found: Tablet not found: f3973dcb94744c4ebe81bdba5358e6ef
I20260619 14:40:01.805270 15324 leader_election.cc:302] T f3973dcb94744c4ebe81bdba5358e6ef P 79e520561a044d52a83a5b8258842cfc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 79e520561a044d52a83a5b8258842cfc; no voters: 23b8ca310b6240ad90478bc027bf937c, 5769a3dda9664950bd099d906adf485b
I20260619 14:40:01.805522 15711 raft_consensus.cc:2726] T f3973dcb94744c4ebe81bdba5358e6ef P 79e520561a044d52a83a5b8258842cfc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 14:40:01.991878 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:01.991634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42505 (local address 127.0.98.1:41149)
0619 14:40:01.991741 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:01.991795 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260619 14:40:02.137733 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:02.137324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42505 (local address 127.0.98.2:41115)
0619 14:40:02.137554 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:02.137607 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":68,"threads_started":1}
I20260619 14:40:02.304335 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:02.304003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42505 (local address 127.0.98.3:41377)
0619 14:40:02.304195 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:02.304244 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":60,"threads_started":1}
I20260619 14:40:02.993167 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:02.992673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42505 (local address 127.0.98.1:47103)
0619 14:40:02.992960 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:02.993027 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":110,"threads_started":1}
I20260619 14:40:03.138935 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:03.138497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42505 (local address 127.0.98.2:42893)
0619 14:40:03.138785 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:03.138843 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":84,"threads_started":1}
I20260619 14:40:03.305318 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:03.304943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42505 (local address 127.0.98.3:44147)
0619 14:40:03.305169 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:03.305221 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":68,"threads_started":1}
I20260619 14:40:03.729789 15721 raft_consensus.cc:482] T f3973dcb94744c4ebe81bdba5358e6ef P 79e520561a044d52a83a5b8258842cfc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:40:03.729918 15721 raft_consensus.cc:504] T f3973dcb94744c4ebe81bdba5358e6ef P 79e520561a044d52a83a5b8258842cfc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5769a3dda9664950bd099d906adf485b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41403 } } peers { permanent_uuid: "23b8ca310b6240ad90478bc027bf937c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40807 } } peers { permanent_uuid: "79e520561a044d52a83a5b8258842cfc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37871 } }
I20260619 14:40:03.730345 15721 leader_election.cc:288] T f3973dcb94744c4ebe81bdba5358e6ef P 79e520561a044d52a83a5b8258842cfc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5769a3dda9664950bd099d906adf485b (127.0.98.2:41403), 23b8ca310b6240ad90478bc027bf937c (127.0.98.3:40807)
W20260619 14:40:03.731061 15326 leader_election.cc:341] T f3973dcb94744c4ebe81bdba5358e6ef P 79e520561a044d52a83a5b8258842cfc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5769a3dda9664950bd099d906adf485b (127.0.98.2:41403): Not found: Tablet not found: f3973dcb94744c4ebe81bdba5358e6ef
W20260619 14:40:03.731160 15324 leader_election.cc:341] T f3973dcb94744c4ebe81bdba5358e6ef P 79e520561a044d52a83a5b8258842cfc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 23b8ca310b6240ad90478bc027bf937c (127.0.98.3:40807): Not found: Tablet not found: f3973dcb94744c4ebe81bdba5358e6ef
I20260619 14:40:03.731245 15324 leader_election.cc:302] T f3973dcb94744c4ebe81bdba5358e6ef P 79e520561a044d52a83a5b8258842cfc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 79e520561a044d52a83a5b8258842cfc; no voters: 23b8ca310b6240ad90478bc027bf937c, 5769a3dda9664950bd099d906adf485b
I20260619 14:40:03.731377 15721 raft_consensus.cc:2726] T f3973dcb94744c4ebe81bdba5358e6ef P 79e520561a044d52a83a5b8258842cfc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 14:40:03.994257 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:03.993864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42505 (local address 127.0.98.1:52953)
0619 14:40:03.994081 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:03.994162 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":69,"threads_started":1}
I20260619 14:40:04.140023 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:04.139622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42505 (local address 127.0.98.2:57453)
0619 14:40:04.139880 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:04.139934 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":84,"threads_started":1}
I20260619 14:40:04.306346 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:04.305969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42505 (local address 127.0.98.3:35745)
0619 14:40:04.306201 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:04.306257 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":84,"threads_started":1}
I20260619 14:40:04.995872 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:04.995146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42505 (local address 127.0.98.1:49999)
0619 14:40:04.995665 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:04.995747 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":79,"threads_started":1}
I20260619 14:40:05.141332 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:05.140708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42505 (local address 127.0.98.2:40781)
0619 14:40:05.141180 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:05.141235 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":132,"threads_started":1}
I20260619 14:40:05.307390 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:05.307003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42505 (local address 127.0.98.3:55415)
0619 14:40:05.307250 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:05.307303 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":82,"threads_started":1}
I20260619 14:40:05.997149 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:05.996674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42505 (local address 127.0.98.1:60465)
0619 14:40:05.996985 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:05.997050 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":122,"threads_started":1}
I20260619 14:40:06.005336 15729 raft_consensus.cc:482] T f3973dcb94744c4ebe81bdba5358e6ef P 79e520561a044d52a83a5b8258842cfc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:40:06.005466 15729 raft_consensus.cc:504] T f3973dcb94744c4ebe81bdba5358e6ef P 79e520561a044d52a83a5b8258842cfc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5769a3dda9664950bd099d906adf485b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41403 } } peers { permanent_uuid: "23b8ca310b6240ad90478bc027bf937c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40807 } } peers { permanent_uuid: "79e520561a044d52a83a5b8258842cfc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37871 } }
I20260619 14:40:06.005820 15729 leader_election.cc:288] T f3973dcb94744c4ebe81bdba5358e6ef P 79e520561a044d52a83a5b8258842cfc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5769a3dda9664950bd099d906adf485b (127.0.98.2:41403), 23b8ca310b6240ad90478bc027bf937c (127.0.98.3:40807)
W20260619 14:40:06.006683 15326 leader_election.cc:341] T f3973dcb94744c4ebe81bdba5358e6ef P 79e520561a044d52a83a5b8258842cfc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5769a3dda9664950bd099d906adf485b (127.0.98.2:41403): Not found: Tablet not found: f3973dcb94744c4ebe81bdba5358e6ef
W20260619 14:40:06.006821 15324 leader_election.cc:341] T f3973dcb94744c4ebe81bdba5358e6ef P 79e520561a044d52a83a5b8258842cfc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 23b8ca310b6240ad90478bc027bf937c (127.0.98.3:40807): Not found: Tablet not found: f3973dcb94744c4ebe81bdba5358e6ef
I20260619 14:40:06.006932 15324 leader_election.cc:302] T f3973dcb94744c4ebe81bdba5358e6ef P 79e520561a044d52a83a5b8258842cfc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 79e520561a044d52a83a5b8258842cfc; no voters: 23b8ca310b6240ad90478bc027bf937c, 5769a3dda9664950bd099d906adf485b
I20260619 14:40:06.007112 15729 raft_consensus.cc:2726] T f3973dcb94744c4ebe81bdba5358e6ef P 79e520561a044d52a83a5b8258842cfc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 14:40:06.142627 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:06.142123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42505 (local address 127.0.98.2:37827)
0619 14:40:06.142422 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:06.142491 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":96,"threads_started":1}
I20260619 14:40:06.308440 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:06.308028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42505 (local address 127.0.98.3:36771)
0619 14:40:06.308286 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:06.308345 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":90,"threads_started":1}
I20260619 14:40:06.998389 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:06.997906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42505 (local address 127.0.98.1:38641)
0619 14:40:06.998194 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:06.998265 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":79,"threads_started":1}
I20260619 14:40:07.143890 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:07.143425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42505 (local address 127.0.98.2:37063)
0619 14:40:07.143729 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:07.143791 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":88,"threads_started":1}
I20260619 14:40:07.309543 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:07.309124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42505 (local address 127.0.98.3:48265)
0619 14:40:07.309396 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:07.309450 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":84,"threads_started":1}
I20260619 14:40:08.000289 15735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:07.999884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42505 (local address 127.0.98.1:34995)
0619 14:40:08.000141 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:08.000198 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":68,"threads_started":1}
I20260619 14:40:08.144939 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:08.144532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42505 (local address 127.0.98.2:40741)
0619 14:40:08.144794 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:08.144847 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":67,"threads_started":1}
I20260619 14:40:08.310598 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:08.310202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42505 (local address 127.0.98.3:45017)
0619 14:40:08.310451 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:08.310505 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":87,"threads_started":1}
I20260619 14:40:08.853336 15738 raft_consensus.cc:482] T f3973dcb94744c4ebe81bdba5358e6ef P 79e520561a044d52a83a5b8258842cfc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:40:08.853466 15738 raft_consensus.cc:504] T f3973dcb94744c4ebe81bdba5358e6ef P 79e520561a044d52a83a5b8258842cfc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5769a3dda9664950bd099d906adf485b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41403 } } peers { permanent_uuid: "23b8ca310b6240ad90478bc027bf937c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40807 } } peers { permanent_uuid: "79e520561a044d52a83a5b8258842cfc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37871 } }
I20260619 14:40:08.853811 15738 leader_election.cc:288] T f3973dcb94744c4ebe81bdba5358e6ef P 79e520561a044d52a83a5b8258842cfc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5769a3dda9664950bd099d906adf485b (127.0.98.2:41403), 23b8ca310b6240ad90478bc027bf937c (127.0.98.3:40807)
W20260619 14:40:08.854557 15324 leader_election.cc:341] T f3973dcb94744c4ebe81bdba5358e6ef P 79e520561a044d52a83a5b8258842cfc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 23b8ca310b6240ad90478bc027bf937c (127.0.98.3:40807): Not found: Tablet not found: f3973dcb94744c4ebe81bdba5358e6ef
W20260619 14:40:08.854674 15326 leader_election.cc:341] T f3973dcb94744c4ebe81bdba5358e6ef P 79e520561a044d52a83a5b8258842cfc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5769a3dda9664950bd099d906adf485b (127.0.98.2:41403): Not found: Tablet not found: f3973dcb94744c4ebe81bdba5358e6ef
I20260619 14:40:08.854775 15326 leader_election.cc:302] T f3973dcb94744c4ebe81bdba5358e6ef P 79e520561a044d52a83a5b8258842cfc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 79e520561a044d52a83a5b8258842cfc; no voters: 23b8ca310b6240ad90478bc027bf937c, 5769a3dda9664950bd099d906adf485b
I20260619 14:40:08.854951 15738 raft_consensus.cc:2726] T f3973dcb94744c4ebe81bdba5358e6ef P 79e520561a044d52a83a5b8258842cfc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 14:40:09.001518 15739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:09.000975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42505 (local address 127.0.98.1:53219)
0619 14:40:09.001310 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:09.001377 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":101,"threads_started":1}
I20260619 14:40:09.146061 15740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:09.145630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42505 (local address 127.0.98.2:46221)
0619 14:40:09.145910 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:09.145966 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":88,"threads_started":1}
I20260619 14:40:09.311779 15741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:09.311385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42505 (local address 127.0.98.3:37155)
0619 14:40:09.311629 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:09.311687 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":73,"threads_started":1}
I20260619 14:40:10.002588 15742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:10.002225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42505 (local address 127.0.98.1:50419)
0619 14:40:10.002436 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:10.002494 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":65,"threads_started":1}
I20260619 14:40:10.147231 15743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:10.146837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42505 (local address 127.0.98.2:36241)
0619 14:40:10.147080 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:10.147135 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":63,"threads_started":1}
I20260619 14:40:10.312928 15744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:10.312508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42505 (local address 127.0.98.3:59163)
0619 14:40:10.312778 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:10.312834 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":70,"threads_started":1}
I20260619 14:40:11.003705 15745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:11.003250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42505 (local address 127.0.98.1:51513)
0619 14:40:11.003521 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:11.003576 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":100,"threads_started":1}
I20260619 14:40:11.148301 15746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:11.147907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42505 (local address 127.0.98.2:58895)
0619 14:40:11.148155 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:11.148206 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":82,"threads_started":1}
I20260619 14:40:11.314009 15747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:11.313579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42505 (local address 127.0.98.3:59201)
0619 14:40:11.313855 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:11.313916 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":83,"threads_started":1}
I20260619 14:40:12.004773 15748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:12.004373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42505 (local address 127.0.98.1:46145)
0619 14:40:12.004596 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:12.004654 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":71,"threads_started":1}
I20260619 14:40:12.149377 15749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:12.148996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42505 (local address 127.0.98.2:51673)
0619 14:40:12.149223 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:12.149276 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":74,"threads_started":1}
I20260619 14:40:12.315110 15750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:12.314685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42505 (local address 127.0.98.3:56943)
0619 14:40:12.314963 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:12.315017 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":107,"threads_started":1}
I20260619 14:40:13.005759 15751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:13.005407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42505 (local address 127.0.98.1:56753)
0619 14:40:13.005616 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:13.005667 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":68,"threads_started":1}
I20260619 14:40:13.150386 15752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:13.150028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42505 (local address 127.0.98.2:48895)
0619 14:40:13.150243 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:13.150294 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":70,"threads_started":1}
I20260619 14:40:13.316133 15753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:13.315749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42505 (local address 127.0.98.3:48281)
0619 14:40:13.315988 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:13.316043 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":73,"threads_started":1}
I20260619 14:40:14.006873 15754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:14.006450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42505 (local address 127.0.98.1:55585)
0619 14:40:14.006718 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:14.006776 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":76,"threads_started":1}
I20260619 14:40:14.151494 15755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:14.151099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42505 (local address 127.0.98.2:44335)
0619 14:40:14.151331 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:14.151395 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":78,"threads_started":1}
I20260619 14:40:14.317332 15756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:14.316919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42505 (local address 127.0.98.3:52007)
0619 14:40:14.317173 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:14.317235 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":89,"threads_started":1}
I20260619 14:40:15.008194 15757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:15.007817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42505 (local address 127.0.98.1:55637)
0619 14:40:15.008048 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:15.008102 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":72,"threads_started":1}
I20260619 14:40:15.152998 15758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:15.152363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42505 (local address 127.0.98.2:36269)
0619 14:40:15.152815 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:15.152880 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":70,"threads_started":1}
I20260619 14:40:15.301205 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 15274
I20260619 14:40:15.306423 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 15441
I20260619 14:40:15.311004 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 15575
2026-06-19T14:40:15Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19153 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:40:15 dist-test-slave-32hj krb5kdc[15761](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:40:15 dist-test-slave-32hj krb5kdc[15761](info): set up 2 sockets
Jun 19 14:40:15 dist-test-slave-32hj krb5kdc[15761](info): commencing operation
krb5kdc: starting...
W20260619 14:40:16.268602 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/test-user.keytab.
Jun 19 14:40:16 dist-test-slave-32hj krb5kdc[15761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:40:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:40:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:40:16.384392 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42853
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42853
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:40:16.492268 15774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:40:16.492534 15774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:40:16.492588 15774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:40:16.496495 15774 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 14:40:16.496547 15774 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 14:40:16.496567 15774 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 14:40:16.496584 15774 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:40:16.496605 15774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:40:16.496629 15774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:40:16.496658 15774 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:40:16.496687 15774 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:40:16.501880 15774 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:40:16.502909 15774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:40:16.503907 15774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:40:16.512475 15779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:40:16.512516 15774 server_base.cc:1034] running on GCE node
W20260619 14:40:16.512463 15780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:40:16.512441 15782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:40:16.513281 15774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:40:16.514374 15774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:40:16.515550 15774 hybrid_clock.cc:648] HybridClock initialized: now 1781880016515524 us; error 42 us; skew 500 ppm
Jun 19 14:40:16 dist-test-slave-32hj krb5kdc[15761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880016, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:40:16.518316 15774 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:40:16.519224 15774 webserver.cc:460] Webserver started at http://127.0.98.62:33523/ using document root <none> and password file <none>
I20260619 14:40:16.519793 15774 fs_manager.cc:359] Metadata directory not provided
I20260619 14:40:16.519860 15774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:40:16.520067 15774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:40:16.522130 15774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/master-0/data/instance:
uuid: "10cf4324bd7d41a39669534d9a7b4e88"
format_stamp: "Formatted at 2026-06-19 14:40:16 on dist-test-slave-32hj"
server_key: "67e38875f0ce57641f1f3b9bc52e7dc5"
server_key_iv: "1a6b7241774355dca89ba4ccad018798"
server_key_version: "encryptionkey@0"
I20260619 14:40:16.522668 15774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/master-0/wal/instance:
uuid: "10cf4324bd7d41a39669534d9a7b4e88"
format_stamp: "Formatted at 2026-06-19 14:40:16 on dist-test-slave-32hj"
server_key: "67e38875f0ce57641f1f3b9bc52e7dc5"
server_key_iv: "1a6b7241774355dca89ba4ccad018798"
server_key_version: "encryptionkey@0"
I20260619 14:40:16.526448 15774 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260619 14:40:16.528986 15789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:40:16.530215 15774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:40:16.530335 15774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/master-0/wal
uuid: "10cf4324bd7d41a39669534d9a7b4e88"
format_stamp: "Formatted at 2026-06-19 14:40:16 on dist-test-slave-32hj"
server_key: "67e38875f0ce57641f1f3b9bc52e7dc5"
server_key_iv: "1a6b7241774355dca89ba4ccad018798"
server_key_version: "encryptionkey@0"
I20260619 14:40:16.530444 15774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:40:16.549373 15774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:40:16.552583 15774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:40:16.552798 15774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:40:16.559345 15774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42853
I20260619 14:40:16.559600 15841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42853 every 8 connection(s)
I20260619 14:40:16.560325 15774 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:40:16.563099 15842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:40:16.568703 15842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88: Bootstrap starting.
I20260619 14:40:16.570631 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 15774
I20260619 14:40:16.570736 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/master-0/wal/instance
I20260619 14:40:16.571163 392 external_mini_cluster.cc:1428] Setting key 4dc9a25fdae47d4e353511b1ef0457ef
I20260619 14:40:16.571419 15842 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88: Neither blocks nor log segments found. Creating new log.
I20260619 14:40:16.572301 15842 log.cc:824] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88: Log is configured to *not* fsync() on all Append() calls
I20260619 14:40:16.574723 15842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88: No bootstrap required, opened a new log
I20260619 14:40:16.579087 15842 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10cf4324bd7d41a39669534d9a7b4e88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42853 } }
I20260619 14:40:16.579329 15842 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:40:16.579367 15842 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10cf4324bd7d41a39669534d9a7b4e88, State: Initialized, Role: FOLLOWER
I20260619 14:40:16.579830 15842 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10cf4324bd7d41a39669534d9a7b4e88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42853 } }
I20260619 14:40:16.579958 15842 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:40:16.580020 15842 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:40:16.580116 15842 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88 [term 0 FOLLOWER]: Advancing to term 1
Jun 19 14:40:16 dist-test-slave-32hj krb5kdc[15761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:40:16.581175 15842 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10cf4324bd7d41a39669534d9a7b4e88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42853 } }
I20260619 14:40:16.581568 15842 leader_election.cc:302] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10cf4324bd7d41a39669534d9a7b4e88; no voters:
I20260619 14:40:16.581911 15842 leader_election.cc:288] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:40:16.582032 15847 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:40:16.582244 15847 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88 [term 1 LEADER]: Becoming Leader. State: Replica: 10cf4324bd7d41a39669534d9a7b4e88, State: Running, Role: LEADER
I20260619 14:40:16.582583 15847 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10cf4324bd7d41a39669534d9a7b4e88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42853 } }
I20260619 14:40:16.583338 15842 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:40:16.583940 15849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "10cf4324bd7d41a39669534d9a7b4e88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10cf4324bd7d41a39669534d9a7b4e88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42853 } } }
I20260619 14:40:16.584066 15849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88 [sys.catalog]: This master's current role is: LEADER
I20260619 14:40:16.584503 15854 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:40:16.584468 15848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10cf4324bd7d41a39669534d9a7b4e88. Latest consensus state: current_term: 1 leader_uuid: "10cf4324bd7d41a39669534d9a7b4e88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10cf4324bd7d41a39669534d9a7b4e88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42853 } } }
I20260619 14:40:16.584668 15848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88 [sys.catalog]: This master's current role is: LEADER
I20260619 14:40:16.587184 15845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:16.572402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53690 (local address 127.0.98.62:42853)
0619 14:40:16.572889 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0619 14:40:16.572906 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 14:40:16.572946 (+ 40us) server_negotiation.cc:407] Connection header received
0619 14:40:16.573811 (+ 865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:40:16.573836 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:40:16.574253 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:40:16.574632 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:40:16.575349 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:40:16.576874 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:40:16.577708 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:40:16.578540 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:40:16.578705 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:40:16.581186 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:40:16.581194 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:40:16.581214 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:40:16.581251 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:40:16.583322 (+ 2071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:40:16.583835 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:40:16.583842 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:40:16.583849 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:40:16.583998 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:40:16.584353 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:40:16.584356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:40:16.584358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:40:16.584960 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:40:16.585175 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0619 14:40:16.585986 (+ 811us) server_negotiation.cc:299] Negotiation successful
0619 14:40:16.586282 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":144,"threads_started":1}
I20260619 14:40:16.587756 15854 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:40:16.593395 15854 catalog_manager.cc:1347] Generated new cluster ID: 305d5b121e2d40a0a884286bc601db9b
I20260619 14:40:16.593477 15854 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:40:16.612110 15854 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:40:16.613471 15854 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:40:16.619143 15854 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88: Generated new TSK 0
I20260619 14:40:16.620046 15854 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 14:40:16.655767 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42853
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:40:16.766698 15868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:40:16.767000 15868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:40:16.767045 15868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:40:16.771162 15868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:40:16.771237 15868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:40:16.771298 15868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:40:16.776940 15868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:40:16.778043 15868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:40:16.779038 15868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:40:16.788968 15876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:40:16.789011 15873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:40:16.788991 15868 server_base.cc:1034] running on GCE node
W20260619 14:40:16.789212 15874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:40:16.789724 15868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:40:16.790416 15868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:40:16.791575 15868 hybrid_clock.cc:648] HybridClock initialized: now 1781880016791556 us; error 31 us; skew 500 ppm
Jun 19 14:40:16 dist-test-slave-32hj krb5kdc[15761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880016, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:40:16.794380 15868 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:40:16.795287 15868 webserver.cc:460] Webserver started at http://127.0.98.1:35437/ using document root <none> and password file <none>
I20260619 14:40:16.795864 15868 fs_manager.cc:359] Metadata directory not provided
I20260619 14:40:16.795935 15868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:40:16.796136 15868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:40:16.798271 15868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-0/data/instance:
uuid: "f68c94798df842509a153535526e1b08"
format_stamp: "Formatted at 2026-06-19 14:40:16 on dist-test-slave-32hj"
server_key: "f34f2e2b1c390ecaf1cf2f124f41a3b3"
server_key_iv: "a80b9e446beec75f72bd2f793c196c6b"
server_key_version: "encryptionkey@0"
I20260619 14:40:16.798834 15868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f68c94798df842509a153535526e1b08"
format_stamp: "Formatted at 2026-06-19 14:40:16 on dist-test-slave-32hj"
server_key: "f34f2e2b1c390ecaf1cf2f124f41a3b3"
server_key_iv: "a80b9e446beec75f72bd2f793c196c6b"
server_key_version: "encryptionkey@0"
I20260619 14:40:16.802649 15868 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:40:16.805239 15883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:40:16.806531 15868 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:40:16.806684 15868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-0/wal
uuid: "f68c94798df842509a153535526e1b08"
format_stamp: "Formatted at 2026-06-19 14:40:16 on dist-test-slave-32hj"
server_key: "f34f2e2b1c390ecaf1cf2f124f41a3b3"
server_key_iv: "a80b9e446beec75f72bd2f793c196c6b"
server_key_version: "encryptionkey@0"
I20260619 14:40:16.806793 15868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:40:16.832082 15868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:40:16.835481 15868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:40:16.835677 15868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:40:16.836293 15868 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:40:16.837359 15868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:40:16.837410 15868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:40:16.837466 15868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:40:16.837493 15868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:40:16.846278 15868 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45973
I20260619 14:40:16.846310 15996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45973 every 8 connection(s)
I20260619 14:40:16.847174 15868 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:40:16.852494 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 15868
I20260619 14:40:16.852618 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:40:16.852941 392 external_mini_cluster.cc:1428] Setting key d9650401361324e0dbe50538656b8999
Jun 19 14:40:16 dist-test-slave-32hj krb5kdc[15761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880016, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:40:16.861614 15845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:16.849121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32853 (local address 127.0.98.62:42853)
0619 14:40:16.849283 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 14:40:16.849288 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:40:16.850349 (+ 1061us) server_negotiation.cc:407] Connection header received
0619 14:40:16.851266 (+ 917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:40:16.851269 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:40:16.851326 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:40:16.851396 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:40:16.852598 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:40:16.853250 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:40:16.854903 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:40:16.855145 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:40:16.855238 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:40:16.858023 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:40:16.858026 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:40:16.858028 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:40:16.858052 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:40:16.859401 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:40:16.859984 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:40:16.859987 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:40:16.859988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:40:16.860054 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:40:16.860431 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:40:16.860435 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:40:16.860436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:40:16.860626 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:40:16.860715 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 14:40:16.861379 (+ 664us) server_negotiation.cc:299] Negotiation successful
0619 14:40:16.861491 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 14:40:16.862753 15999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:16.849527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42853 (local address 127.0.98.1:32853)
0619 14:40:16.850189 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:16.850233 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 14:40:16.851049 (+ 816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:40:16.851575 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:40:16.851582 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:40:16.851952 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:40:16.852459 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:40:16.852472 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:40:16.853368 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:40:16.853371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:40:16.854719 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:40:16.854734 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:40:16.855294 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:40:16.855299 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:40:16.855457 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:40:16.856090 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:40:16.856111 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:40:16.857854 (+ 1743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:40:16.859515 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:40:16.859522 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:40:16.859541 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:40:16.859898 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:40:16.860126 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:40:16.860129 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:40:16.860131 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:40:16.860313 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:40:16.860758 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:40:16.860766 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:40:16.861105 (+ 339us) client_negotiation.cc:769] Sending connection context
0619 14:40:16.861329 (+ 224us) client_negotiation.cc:240] Negotiation successful
0619 14:40:16.861682 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":118,"threads_started":1}
I20260619 14:40:16.863988 15997 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42853
I20260619 14:40:16.864202 15997 heartbeater.cc:464] Registering TS with master...
I20260619 14:40:16.864748 15997 heartbeater.cc:511] Master 127.0.98.62:42853 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 14:40:16.866194 15806 ts_manager.cc:194] Registered new tserver with Master: f68c94798df842509a153535526e1b08 (127.0.98.1:45973)
I20260619 14:40:16.867308 15806 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:32853
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 14:40:16.880087 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42853
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:40:16.989456 16002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:40:16.989720 16002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:40:16.989766 16002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:40:16.994154 16002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:40:16.994230 16002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:40:16.994313 16002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:40:17.001259 16002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:40:17.002374 16002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:40:17.003407 16002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:40:17.013031 16007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:40:17.013044 16008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:40:17.013044 16010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:40:17.013280 16002 server_base.cc:1034] running on GCE node
I20260619 14:40:17.013692 16002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:40:17.014346 16002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:40:17.015496 16002 hybrid_clock.cc:648] HybridClock initialized: now 1781880017015470 us; error 38 us; skew 500 ppm
Jun 19 14:40:17 dist-test-slave-32hj krb5kdc[15761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880017, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:40:17.018040 16002 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:40:17.018882 16002 webserver.cc:460] Webserver started at http://127.0.98.2:38117/ using document root <none> and password file <none>
I20260619 14:40:17.019411 16002 fs_manager.cc:359] Metadata directory not provided
I20260619 14:40:17.019476 16002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:40:17.019666 16002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:40:17.021696 16002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-1/data/instance:
uuid: "f7d3440928d640e6ad9f508d12cadd33"
format_stamp: "Formatted at 2026-06-19 14:40:17 on dist-test-slave-32hj"
server_key: "ac943e5cab4391b4dc461e33a05a02ce"
server_key_iv: "e2ec0d1056d386cfa8c0ba2c14146d7c"
server_key_version: "encryptionkey@0"
I20260619 14:40:17.022198 16002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f7d3440928d640e6ad9f508d12cadd33"
format_stamp: "Formatted at 2026-06-19 14:40:17 on dist-test-slave-32hj"
server_key: "ac943e5cab4391b4dc461e33a05a02ce"
server_key_iv: "e2ec0d1056d386cfa8c0ba2c14146d7c"
server_key_version: "encryptionkey@0"
I20260619 14:40:17.025728 16002 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:40:17.028029 16017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:40:17.029309 16002 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:40:17.029424 16002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-1/wal
uuid: "f7d3440928d640e6ad9f508d12cadd33"
format_stamp: "Formatted at 2026-06-19 14:40:17 on dist-test-slave-32hj"
server_key: "ac943e5cab4391b4dc461e33a05a02ce"
server_key_iv: "e2ec0d1056d386cfa8c0ba2c14146d7c"
server_key_version: "encryptionkey@0"
I20260619 14:40:17.029515 16002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:40:17.047892 16002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:40:17.051406 16002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:40:17.051602 16002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:40:17.052227 16002 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:40:17.053102 16002 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:40:17.053149 16002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:40:17.053200 16002 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:40:17.053229 16002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:40:17.061981 16002 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42139
I20260619 14:40:17.062026 16130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42139 every 8 connection(s)
I20260619 14:40:17.062911 16002 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:40:17.066025 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 16002
I20260619 14:40:17.066149 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:40:17.066485 392 external_mini_cluster.cc:1428] Setting key 86be14768169bb9ef66c34198a7028e4
Jun 19 14:40:17 dist-test-slave-32hj krb5kdc[15761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880017, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 14:40:17.078264 15845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:17.064781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57907 (local address 127.0.98.62:42853)
0619 14:40:17.064932 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 14:40:17.064936 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:40:17.065823 (+ 887us) server_negotiation.cc:407] Connection header received
0619 14:40:17.067405 (+ 1582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:40:17.067409 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:40:17.067461 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:40:17.067536 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:40:17.069157 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:40:17.069838 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:40:17.071038 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:40:17.071356 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:40:17.071457 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:40:17.074290 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:40:17.074293 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:40:17.074296 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:40:17.074323 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:40:17.075785 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:40:17.076415 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:40:17.076419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:40:17.076420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:40:17.076512 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:40:17.076979 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:40:17.076982 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:40:17.076983 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:40:17.077180 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:40:17.077289 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 14:40:17.077905 (+ 616us) server_negotiation.cc:299] Negotiation successful
0619 14:40:17.078088 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 14:40:17.079070 16133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:17.065159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42853 (local address 127.0.98.2:57907)
0619 14:40:17.065640 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:17.065686 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 14:40:17.067181 (+ 1495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:40:17.067738 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:40:17.067748 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:40:17.068244 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:40:17.068982 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:40:17.069001 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:40:17.069981 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:40:17.069984 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:40:17.070913 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:40:17.070923 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:40:17.071509 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:40:17.071512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:40:17.071667 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:40:17.072333 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:40:17.072350 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:40:17.074077 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:40:17.075914 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:40:17.075919 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:40:17.075933 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:40:17.076252 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:40:17.076654 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:40:17.076657 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:40:17.076658 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:40:17.076867 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:40:17.077341 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:40:17.077348 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:40:17.077617 (+ 269us) client_negotiation.cc:769] Sending connection context
0619 14:40:17.077811 (+ 194us) client_negotiation.cc:240] Negotiation successful
0619 14:40:17.078123 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":119,"threads_started":1}
I20260619 14:40:17.080372 16131 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42853
I20260619 14:40:17.080621 16131 heartbeater.cc:464] Registering TS with master...
I20260619 14:40:17.081149 16131 heartbeater.cc:511] Master 127.0.98.62:42853 requested a full tablet report, sending...
I20260619 14:40:17.082270 15806 ts_manager.cc:194] Registered new tserver with Master: f7d3440928d640e6ad9f508d12cadd33 (127.0.98.2:42139)
I20260619 14:40:17.082839 15806 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57907
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 14:40:17.096495 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42853
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:40:17.208997 16136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:40:17.209381 16136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:40:17.209450 16136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:40:17.214061 16136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:40:17.214133 16136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:40:17.214195 16136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:40:17.219676 16136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:40:17.220775 16136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:40:17.221789 16136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:40:17.231202 16142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:40:17.231212 16141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:40:17.231241 16144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:40:17.231567 16136 server_base.cc:1034] running on GCE node
I20260619 14:40:17.232021 16136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:40:17.232668 16136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:40:17.233850 16136 hybrid_clock.cc:648] HybridClock initialized: now 1781880017233828 us; error 33 us; skew 500 ppm
Jun 19 14:40:17 dist-test-slave-32hj krb5kdc[15761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880017, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:40:17.236449 16136 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:40:17.237345 16136 webserver.cc:460] Webserver started at http://127.0.98.3:39331/ using document root <none> and password file <none>
I20260619 14:40:17.237870 16136 fs_manager.cc:359] Metadata directory not provided
I20260619 14:40:17.237926 16136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:40:17.238093 16136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:40:17.240141 16136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-2/data/instance:
uuid: "be01008d6c3647208fe2f0acebe5742d"
format_stamp: "Formatted at 2026-06-19 14:40:17 on dist-test-slave-32hj"
server_key: "71125b9862263840417123ad68383cce"
server_key_iv: "297aad49cc22b37b4f186ebe45e21c31"
server_key_version: "encryptionkey@0"
I20260619 14:40:17.240648 16136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-2/wal/instance:
uuid: "be01008d6c3647208fe2f0acebe5742d"
format_stamp: "Formatted at 2026-06-19 14:40:17 on dist-test-slave-32hj"
server_key: "71125b9862263840417123ad68383cce"
server_key_iv: "297aad49cc22b37b4f186ebe45e21c31"
server_key_version: "encryptionkey@0"
I20260619 14:40:17.244436 16136 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 14:40:17.246856 16151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:40:17.248021 16136 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:40:17.248124 16136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-2/wal
uuid: "be01008d6c3647208fe2f0acebe5742d"
format_stamp: "Formatted at 2026-06-19 14:40:17 on dist-test-slave-32hj"
server_key: "71125b9862263840417123ad68383cce"
server_key_iv: "297aad49cc22b37b4f186ebe45e21c31"
server_key_version: "encryptionkey@0"
I20260619 14:40:17.248230 16136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:40:17.267998 16136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:40:17.271081 16136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:40:17.271257 16136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:40:17.271858 16136 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:40:17.272707 16136 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:40:17.272794 16136 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:40:17.272847 16136 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:40:17.272876 16136 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:40:17.281517 16136 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35985
I20260619 14:40:17.281545 16264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35985 every 8 connection(s)
I20260619 14:40:17.282505 16136 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:40:17.283025 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 16136
I20260619 14:40:17.283180 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:40:17.283460 392 external_mini_cluster.cc:1428] Setting key 5b3871b2480c126a6b5b0987421216e4
Jun 19 14:40:17 dist-test-slave-32hj krb5kdc[15761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880017, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:40:17.295981 15845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:17.284423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58645 (local address 127.0.98.62:42853)
0619 14:40:17.284549 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 14:40:17.284554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:40:17.285973 (+ 1419us) server_negotiation.cc:407] Connection header received
0619 14:40:17.286846 (+ 873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:40:17.286848 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:40:17.286893 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:40:17.286963 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:40:17.288105 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:40:17.288633 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:40:17.289776 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:40:17.290001 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:40:17.290074 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:40:17.292533 (+ 2459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:40:17.292537 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:40:17.292539 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:40:17.292564 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:40:17.293921 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:40:17.294451 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:40:17.294454 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:40:17.294455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:40:17.294532 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:40:17.294905 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:40:17.294908 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:40:17.294909 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:40:17.295093 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:40:17.295175 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 14:40:17.295715 (+ 540us) server_negotiation.cc:299] Negotiation successful
0619 14:40:17.295853 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 14:40:17.296984 16267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:17.285022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42853 (local address 127.0.98.3:58645)
0619 14:40:17.285836 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:17.285884 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 14:40:17.286636 (+ 752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:40:17.287099 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:40:17.287106 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:40:17.287486 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:40:17.287978 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:40:17.287993 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:40:17.288755 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:40:17.288758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:40:17.289674 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:40:17.289682 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:40:17.290153 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:40:17.290156 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:40:17.290307 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:40:17.290860 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:40:17.290876 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:40:17.292360 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:40:17.294043 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:40:17.294048 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:40:17.294080 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:40:17.294357 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:40:17.294657 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:40:17.294660 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:40:17.294663 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:40:17.294814 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:40:17.295221 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:40:17.295227 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:40:17.295489 (+ 262us) client_negotiation.cc:769] Sending connection context
0619 14:40:17.295657 (+ 168us) client_negotiation.cc:240] Negotiation successful
0619 14:40:17.295974 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":173,"threads_started":1}
I20260619 14:40:17.297973 16265 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42853
I20260619 14:40:17.298195 16265 heartbeater.cc:464] Registering TS with master...
I20260619 14:40:17.298666 16265 heartbeater.cc:511] Master 127.0.98.62:42853 requested a full tablet report, sending...
I20260619 14:40:17.299602 15806 ts_manager.cc:194] Registered new tserver with Master: be01008d6c3647208fe2f0acebe5742d (127.0.98.3:35985)
I20260619 14:40:17.300154 15806 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58645
I20260619 14:40:17.309222 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:40:17.318253 15845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:17.311146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53702 (local address 127.0.98.62:42853)
0619 14:40:17.311275 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 14:40:17.311279 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:40:17.311449 (+ 170us) server_negotiation.cc:407] Connection header received
0619 14:40:17.311620 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:40:17.311623 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:40:17.311666 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:40:17.311726 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:40:17.312407 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:40:17.312982 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:40:17.313795 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:40:17.314009 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:40:17.314093 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:40:17.315399 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:40:17.315402 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:40:17.315404 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:40:17.315428 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:40:17.316696 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:40:17.317089 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:40:17.317093 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:40:17.317094 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:40:17.317171 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:40:17.317404 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:40:17.317407 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:40:17.317409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:40:17.317628 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:40:17.317751 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 14:40:17.317984 (+ 233us) server_negotiation.cc:299] Negotiation successful
0619 14:40:17.318100 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 14:40:17.321144 15806 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53702:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:40:17.324095 15806 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:40:17.338101 16277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:17.332339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35004 (local address 127.0.98.1:45973)
0619 14:40:17.333138 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0619 14:40:17.333152 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:40:17.333175 (+ 23us) server_negotiation.cc:407] Connection header received
0619 14:40:17.333258 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:40:17.333264 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:40:17.333502 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:40:17.334028 (+ 526us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:40:17.334187 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:40:17.335421 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:40:17.336821 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:40:17.337434 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:40:17.337535 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:40:17.337689 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 14:40:17.337869 (+ 180us) server_negotiation.cc:299] Negotiation successful
0619 14:40:17.337967 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":692,"thread_start_us":90,"threads_started":1}
I20260619 14:40:17.338092 16276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:17.332167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45973 (local address 127.0.0.1:35004)
0619 14:40:17.332537 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:17.332594 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 14:40:17.332752 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:40:17.333668 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:40:17.333674 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:40:17.333718 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:40:17.333878 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:40:17.333886 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:40:17.335569 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:40:17.335573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:40:17.336649 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:40:17.336658 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:40:17.337549 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:40:17.337552 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:40:17.337666 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:40:17.337705 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 14:40:17.337809 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 14:40:17.337938 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":76,"thread_start_us":115,"threads_started":1}
I20260619 14:40:17.338493 16278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:17.332393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35985 (local address 127.0.0.1:40938)
0619 14:40:17.332777 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:17.332789 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:40:17.332863 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:40:17.334278 (+ 1415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:40:17.334281 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:40:17.334298 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:40:17.334383 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:40:17.334387 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:40:17.335433 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:40:17.335436 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:40:17.336895 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:40:17.336906 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:40:17.338138 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:40:17.338142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:40:17.338227 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:40:17.338248 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 14:40:17.338320 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 14:40:17.338376 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":125,"thread_start_us":81,"threads_started":1}
I20260619 14:40:17.338493 16279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:17.332958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42139 (local address 127.0.0.1:44202)
0619 14:40:17.333170 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:17.333182 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:40:17.333264 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:40:17.333935 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:40:17.333938 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:40:17.333955 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:40:17.334028 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:40:17.334033 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:40:17.335502 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:40:17.335504 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:40:17.336895 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:40:17.336906 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:40:17.337972 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:40:17.337975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:40:17.338067 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:40:17.338089 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 14:40:17.338137 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 14:40:17.338346 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":96,"thread_start_us":92,"threads_started":1}
I20260619 14:40:17.338917 16281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:17.333250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40938 (local address 127.0.98.3:35985)
0619 14:40:17.333655 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0619 14:40:17.333667 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:40:17.333698 (+ 31us) server_negotiation.cc:407] Connection header received
0619 14:40:17.333774 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:40:17.333779 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:40:17.333989 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:40:17.334132 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:40:17.334530 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:40:17.335319 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:40:17.337032 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:40:17.337839 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:40:17.337941 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:40:17.338085 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 14:40:17.338681 (+ 596us) server_negotiation.cc:299] Negotiation successful
0619 14:40:17.338783 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":127,"threads_started":1}
I20260619 14:40:17.339439 16280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:17.333118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44202 (local address 127.0.98.2:42139)
0619 14:40:17.333479 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0619 14:40:17.333490 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:40:17.333511 (+ 21us) server_negotiation.cc:407] Connection header received
0619 14:40:17.333581 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:40:17.333586 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:40:17.333781 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:40:17.334153 (+ 372us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:40:17.334301 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:40:17.335378 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:40:17.337032 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:40:17.337849 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:40:17.339023 (+ 1174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:40:17.339174 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 14:40:17.339219 (+ 45us) server_negotiation.cc:299] Negotiation successful
0619 14:40:17.339308 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":133,"threads_started":1}
I20260619 14:40:17.341403 16199 tablet_service.cc:1467] Processing CreateTablet for tablet 02c60ec285f84d08a38724bcc9add308 (DEFAULT_TABLE table=test-table [id=16e84011baba433593e7a61a0e2ae8fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:40:17.341403 15931 tablet_service.cc:1467] Processing CreateTablet for tablet 02c60ec285f84d08a38724bcc9add308 (DEFAULT_TABLE table=test-table [id=16e84011baba433593e7a61a0e2ae8fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:40:17.341648 16065 tablet_service.cc:1467] Processing CreateTablet for tablet 02c60ec285f84d08a38724bcc9add308 (DEFAULT_TABLE table=test-table [id=16e84011baba433593e7a61a0e2ae8fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:40:17.342500 16065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02c60ec285f84d08a38724bcc9add308. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:40:17.342522 15931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02c60ec285f84d08a38724bcc9add308. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:40:17.342526 16199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02c60ec285f84d08a38724bcc9add308. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:40:17.349431 16284 tablet_bootstrap.cc:492] T 02c60ec285f84d08a38724bcc9add308 P be01008d6c3647208fe2f0acebe5742d: Bootstrap starting.
I20260619 14:40:17.350302 16283 tablet_bootstrap.cc:492] T 02c60ec285f84d08a38724bcc9add308 P f68c94798df842509a153535526e1b08: Bootstrap starting.
I20260619 14:40:17.348688 16282 tablet_bootstrap.cc:492] T 02c60ec285f84d08a38724bcc9add308 P f7d3440928d640e6ad9f508d12cadd33: Bootstrap starting.
I20260619 14:40:17.351804 16284 tablet_bootstrap.cc:654] T 02c60ec285f84d08a38724bcc9add308 P be01008d6c3647208fe2f0acebe5742d: Neither blocks nor log segments found. Creating new log.
I20260619 14:40:17.352162 16283 tablet_bootstrap.cc:654] T 02c60ec285f84d08a38724bcc9add308 P f68c94798df842509a153535526e1b08: Neither blocks nor log segments found. Creating new log.
I20260619 14:40:17.353053 16282 tablet_bootstrap.cc:654] T 02c60ec285f84d08a38724bcc9add308 P f7d3440928d640e6ad9f508d12cadd33: Neither blocks nor log segments found. Creating new log.
I20260619 14:40:17.353147 16284 log.cc:824] T 02c60ec285f84d08a38724bcc9add308 P be01008d6c3647208fe2f0acebe5742d: Log is configured to *not* fsync() on all Append() calls
I20260619 14:40:17.353223 16283 log.cc:824] T 02c60ec285f84d08a38724bcc9add308 P f68c94798df842509a153535526e1b08: Log is configured to *not* fsync() on all Append() calls
I20260619 14:40:17.353804 16282 log.cc:824] T 02c60ec285f84d08a38724bcc9add308 P f7d3440928d640e6ad9f508d12cadd33: Log is configured to *not* fsync() on all Append() calls
I20260619 14:40:17.355206 16283 tablet_bootstrap.cc:492] T 02c60ec285f84d08a38724bcc9add308 P f68c94798df842509a153535526e1b08: No bootstrap required, opened a new log
I20260619 14:40:17.355235 16284 tablet_bootstrap.cc:492] T 02c60ec285f84d08a38724bcc9add308 P be01008d6c3647208fe2f0acebe5742d: No bootstrap required, opened a new log
I20260619 14:40:17.355419 16283 ts_tablet_manager.cc:1397] T 02c60ec285f84d08a38724bcc9add308 P f68c94798df842509a153535526e1b08: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:40:17.355423 16284 ts_tablet_manager.cc:1397] T 02c60ec285f84d08a38724bcc9add308 P be01008d6c3647208fe2f0acebe5742d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 14:40:17.356171 16282 tablet_bootstrap.cc:492] T 02c60ec285f84d08a38724bcc9add308 P f7d3440928d640e6ad9f508d12cadd33: No bootstrap required, opened a new log
I20260619 14:40:17.356348 16282 ts_tablet_manager.cc:1397] T 02c60ec285f84d08a38724bcc9add308 P f7d3440928d640e6ad9f508d12cadd33: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20260619 14:40:17.358445 16283 raft_consensus.cc:348] T 02c60ec285f84d08a38724bcc9add308 P f68c94798df842509a153535526e1b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68c94798df842509a153535526e1b08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45973 } } peers { permanent_uuid: "be01008d6c3647208fe2f0acebe5742d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35985 } } peers { permanent_uuid: "f7d3440928d640e6ad9f508d12cadd33" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42139 } }
I20260619 14:40:17.358673 16283 raft_consensus.cc:374] T 02c60ec285f84d08a38724bcc9add308 P f68c94798df842509a153535526e1b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:40:17.358723 16283 raft_consensus.cc:729] T 02c60ec285f84d08a38724bcc9add308 P f68c94798df842509a153535526e1b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f68c94798df842509a153535526e1b08, State: Initialized, Role: FOLLOWER
I20260619 14:40:17.359150 16283 consensus_queue.cc:260] T 02c60ec285f84d08a38724bcc9add308 P f68c94798df842509a153535526e1b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68c94798df842509a153535526e1b08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45973 } } peers { permanent_uuid: "be01008d6c3647208fe2f0acebe5742d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35985 } } peers { permanent_uuid: "f7d3440928d640e6ad9f508d12cadd33" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42139 } }
I20260619 14:40:17.359212 16284 raft_consensus.cc:348] T 02c60ec285f84d08a38724bcc9add308 P be01008d6c3647208fe2f0acebe5742d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68c94798df842509a153535526e1b08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45973 } } peers { permanent_uuid: "be01008d6c3647208fe2f0acebe5742d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35985 } } peers { permanent_uuid: "f7d3440928d640e6ad9f508d12cadd33" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42139 } }
I20260619 14:40:17.359498 16284 raft_consensus.cc:374] T 02c60ec285f84d08a38724bcc9add308 P be01008d6c3647208fe2f0acebe5742d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:40:17.359557 16284 raft_consensus.cc:729] T 02c60ec285f84d08a38724bcc9add308 P be01008d6c3647208fe2f0acebe5742d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be01008d6c3647208fe2f0acebe5742d, State: Initialized, Role: FOLLOWER
I20260619 14:40:17.359922 15997 heartbeater.cc:503] Master 127.0.98.62:42853 was elected leader, sending a full tablet report...
I20260619 14:40:17.360024 16284 consensus_queue.cc:260] T 02c60ec285f84d08a38724bcc9add308 P be01008d6c3647208fe2f0acebe5742d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68c94798df842509a153535526e1b08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45973 } } peers { permanent_uuid: "be01008d6c3647208fe2f0acebe5742d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35985 } } peers { permanent_uuid: "f7d3440928d640e6ad9f508d12cadd33" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42139 } }
I20260619 14:40:17.359965 16282 raft_consensus.cc:348] T 02c60ec285f84d08a38724bcc9add308 P f7d3440928d640e6ad9f508d12cadd33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68c94798df842509a153535526e1b08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45973 } } peers { permanent_uuid: "be01008d6c3647208fe2f0acebe5742d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35985 } } peers { permanent_uuid: "f7d3440928d640e6ad9f508d12cadd33" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42139 } }
I20260619 14:40:17.360210 16282 raft_consensus.cc:374] T 02c60ec285f84d08a38724bcc9add308 P f7d3440928d640e6ad9f508d12cadd33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:40:17.360266 16282 raft_consensus.cc:729] T 02c60ec285f84d08a38724bcc9add308 P f7d3440928d640e6ad9f508d12cadd33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7d3440928d640e6ad9f508d12cadd33, State: Initialized, Role: FOLLOWER
I20260619 14:40:17.360674 16282 consensus_queue.cc:260] T 02c60ec285f84d08a38724bcc9add308 P f7d3440928d640e6ad9f508d12cadd33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68c94798df842509a153535526e1b08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45973 } } peers { permanent_uuid: "be01008d6c3647208fe2f0acebe5742d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35985 } } peers { permanent_uuid: "f7d3440928d640e6ad9f508d12cadd33" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42139 } }
I20260619 14:40:17.361073 16284 ts_tablet_manager.cc:1428] T 02c60ec285f84d08a38724bcc9add308 P be01008d6c3647208fe2f0acebe5742d: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 14:40:17.361320 16283 ts_tablet_manager.cc:1428] T 02c60ec285f84d08a38724bcc9add308 P f68c94798df842509a153535526e1b08: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 14:40:17.361423 16265 heartbeater.cc:503] Master 127.0.98.62:42853 was elected leader, sending a full tablet report...
I20260619 14:40:17.362233 16282 ts_tablet_manager.cc:1428] T 02c60ec285f84d08a38724bcc9add308 P f7d3440928d640e6ad9f508d12cadd33: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260619 14:40:17.362579 16131 heartbeater.cc:503] Master 127.0.98.62:42853 was elected leader, sending a full tablet report...
I20260619 14:40:17.379963 16290 raft_consensus.cc:482] T 02c60ec285f84d08a38724bcc9add308 P f7d3440928d640e6ad9f508d12cadd33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:40:17.380172 16290 raft_consensus.cc:504] T 02c60ec285f84d08a38724bcc9add308 P f7d3440928d640e6ad9f508d12cadd33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68c94798df842509a153535526e1b08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45973 } } peers { permanent_uuid: "be01008d6c3647208fe2f0acebe5742d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35985 } } peers { permanent_uuid: "f7d3440928d640e6ad9f508d12cadd33" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42139 } }
I20260619 14:40:17.381392 16290 leader_election.cc:288] T 02c60ec285f84d08a38724bcc9add308 P f7d3440928d640e6ad9f508d12cadd33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f68c94798df842509a153535526e1b08 (127.0.98.1:45973), be01008d6c3647208fe2f0acebe5742d (127.0.98.3:35985)
I20260619 14:40:17.384904 16133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:17.381595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45973 (local address 127.0.98.2:55741)
0619 14:40:17.381711 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:17.381726 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:40:17.381813 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:40:17.382061 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:40:17.382064 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:40:17.382081 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:40:17.382195 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:40:17.382200 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:40:17.382917 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:40:17.382920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:40:17.383832 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:40:17.383841 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:40:17.384470 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:40:17.384474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:40:17.384576 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:40:17.384594 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 14:40:17.384659 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 14:40:17.384765 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":32,"spinlock_wait_cycles":768}
I20260619 14:40:17.384912 16277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:17.381700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55741 (local address 127.0.98.1:45973)
0619 14:40:17.381841 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 14:40:17.381846 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:40:17.381862 (+ 16us) server_negotiation.cc:407] Connection header received
0619 14:40:17.381916 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:40:17.381919 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:40:17.381965 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:40:17.382039 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:40:17.382310 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:40:17.382817 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:40:17.383940 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:40:17.384367 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:40:17.384476 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:40:17.384540 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 14:40:17.384681 (+ 141us) server_negotiation.cc:299] Negotiation successful
0619 14:40:17.384815 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 14:40:17.385098 16291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:17.381652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35985 (local address 127.0.98.2:55751)
0619 14:40:17.382006 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:17.382017 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:40:17.382100 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:40:17.382320 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:40:17.382323 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:40:17.382337 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:40:17.382437 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:40:17.382442 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:40:17.383097 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:40:17.383101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:40:17.383998 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:40:17.384008 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:40:17.384673 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:40:17.384676 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:40:17.384800 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:40:17.384822 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 14:40:17.384885 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 14:40:17.384950 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":14,"thread_start_us":138,"threads_started":1}
I20260619 14:40:17.385098 16281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:17.381872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55751 (local address 127.0.98.3:35985)
0619 14:40:17.381986 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 14:40:17.381990 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:40:17.382082 (+ 92us) server_negotiation.cc:407] Connection header received
0619 14:40:17.382192 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:40:17.382195 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:40:17.382235 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:40:17.382293 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:40:17.382529 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:40:17.383001 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:40:17.384110 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:40:17.384571 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:40:17.384660 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:40:17.384752 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 14:40:17.384910 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 14:40:17.384972 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 14:40:17.385667 16219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02c60ec285f84d08a38724bcc9add308" candidate_uuid: "f7d3440928d640e6ad9f508d12cadd33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be01008d6c3647208fe2f0acebe5742d" is_pre_election: true
I20260619 14:40:17.385679 15951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02c60ec285f84d08a38724bcc9add308" candidate_uuid: "f7d3440928d640e6ad9f508d12cadd33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f68c94798df842509a153535526e1b08" is_pre_election: true
I20260619 14:40:17.385934 15951 raft_consensus.cc:2454] T 02c60ec285f84d08a38724bcc9add308 P f68c94798df842509a153535526e1b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7d3440928d640e6ad9f508d12cadd33 in term 0.
I20260619 14:40:17.385941 16219 raft_consensus.cc:2454] T 02c60ec285f84d08a38724bcc9add308 P be01008d6c3647208fe2f0acebe5742d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7d3440928d640e6ad9f508d12cadd33 in term 0.
I20260619 14:40:17.386401 16018 leader_election.cc:302] T 02c60ec285f84d08a38724bcc9add308 P f7d3440928d640e6ad9f508d12cadd33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be01008d6c3647208fe2f0acebe5742d, f7d3440928d640e6ad9f508d12cadd33; no voters:
I20260619 14:40:17.386674 16290 raft_consensus.cc:2781] T 02c60ec285f84d08a38724bcc9add308 P f7d3440928d640e6ad9f508d12cadd33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:40:17.386749 16290 raft_consensus.cc:482] T 02c60ec285f84d08a38724bcc9add308 P f7d3440928d640e6ad9f508d12cadd33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:40:17.386778 16290 raft_consensus.cc:3037] T 02c60ec285f84d08a38724bcc9add308 P f7d3440928d640e6ad9f508d12cadd33 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:40:17.387811 16290 raft_consensus.cc:504] T 02c60ec285f84d08a38724bcc9add308 P f7d3440928d640e6ad9f508d12cadd33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68c94798df842509a153535526e1b08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45973 } } peers { permanent_uuid: "be01008d6c3647208fe2f0acebe5742d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35985 } } peers { permanent_uuid: "f7d3440928d640e6ad9f508d12cadd33" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42139 } }
I20260619 14:40:17.388151 16290 leader_election.cc:288] T 02c60ec285f84d08a38724bcc9add308 P f7d3440928d640e6ad9f508d12cadd33 [CANDIDATE]: Term 1 election: Requested vote from peers f68c94798df842509a153535526e1b08 (127.0.98.1:45973), be01008d6c3647208fe2f0acebe5742d (127.0.98.3:35985)
I20260619 14:40:17.388484 16219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02c60ec285f84d08a38724bcc9add308" candidate_uuid: "f7d3440928d640e6ad9f508d12cadd33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be01008d6c3647208fe2f0acebe5742d"
I20260619 14:40:17.388588 16219 raft_consensus.cc:3037] T 02c60ec285f84d08a38724bcc9add308 P be01008d6c3647208fe2f0acebe5742d [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:40:17.388598 15951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02c60ec285f84d08a38724bcc9add308" candidate_uuid: "f7d3440928d640e6ad9f508d12cadd33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f68c94798df842509a153535526e1b08"
I20260619 14:40:17.388739 15951 raft_consensus.cc:3037] T 02c60ec285f84d08a38724bcc9add308 P f68c94798df842509a153535526e1b08 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:40:17.389627 16219 raft_consensus.cc:2454] T 02c60ec285f84d08a38724bcc9add308 P be01008d6c3647208fe2f0acebe5742d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7d3440928d640e6ad9f508d12cadd33 in term 1.
I20260619 14:40:17.389758 15951 raft_consensus.cc:2454] T 02c60ec285f84d08a38724bcc9add308 P f68c94798df842509a153535526e1b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7d3440928d640e6ad9f508d12cadd33 in term 1.
I20260619 14:40:17.389902 16018 leader_election.cc:302] T 02c60ec285f84d08a38724bcc9add308 P f7d3440928d640e6ad9f508d12cadd33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be01008d6c3647208fe2f0acebe5742d, f7d3440928d640e6ad9f508d12cadd33; no voters:
I20260619 14:40:17.390057 16290 raft_consensus.cc:2781] T 02c60ec285f84d08a38724bcc9add308 P f7d3440928d640e6ad9f508d12cadd33 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:40:17.390272 16290 raft_consensus.cc:686] T 02c60ec285f84d08a38724bcc9add308 P f7d3440928d640e6ad9f508d12cadd33 [term 1 LEADER]: Becoming Leader. State: Replica: f7d3440928d640e6ad9f508d12cadd33, State: Running, Role: LEADER
I20260619 14:40:17.390633 16290 consensus_queue.cc:237] T 02c60ec285f84d08a38724bcc9add308 P f7d3440928d640e6ad9f508d12cadd33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68c94798df842509a153535526e1b08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45973 } } peers { permanent_uuid: "be01008d6c3647208fe2f0acebe5742d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35985 } } peers { permanent_uuid: "f7d3440928d640e6ad9f508d12cadd33" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42139 } }
I20260619 14:40:17.393635 15802 catalog_manager.cc:5495] T 02c60ec285f84d08a38724bcc9add308 P f7d3440928d640e6ad9f508d12cadd33 reported cstate change: term changed from 0 to 1, leader changed from <none> to f7d3440928d640e6ad9f508d12cadd33 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "f7d3440928d640e6ad9f508d12cadd33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68c94798df842509a153535526e1b08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be01008d6c3647208fe2f0acebe5742d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7d3440928d640e6ad9f508d12cadd33" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42139 } health_report { overall_health: HEALTHY } } }
I20260619 14:40:17.410312 16280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:17.406892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44206 (local address 127.0.98.2:42139)
0619 14:40:17.407006 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 14:40:17.407010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:40:17.407023 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:40:17.407117 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:40:17.407120 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:40:17.407174 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:40:17.407261 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:40:17.407491 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:40:17.408003 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:40:17.408822 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:40:17.409028 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:40:17.409102 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:40:17.409389 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:40:17.409942 (+ 553us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:40:17.410053 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 14:40:17.410121 (+ 68us) server_negotiation.cc:299] Negotiation successful
0619 14:40:17.410173 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 14:40:17.416059 16219 raft_consensus.cc:1261] T 02c60ec285f84d08a38724bcc9add308 P be01008d6c3647208fe2f0acebe5742d [term 1 FOLLOWER]: Refusing update from remote peer f7d3440928d640e6ad9f508d12cadd33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:40:17.416059 15951 raft_consensus.cc:1261] T 02c60ec285f84d08a38724bcc9add308 P f68c94798df842509a153535526e1b08 [term 1 FOLLOWER]: Refusing update from remote peer f7d3440928d640e6ad9f508d12cadd33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:40:17.416859 16292 consensus_queue.cc:1035] T 02c60ec285f84d08a38724bcc9add308 P f7d3440928d640e6ad9f508d12cadd33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f68c94798df842509a153535526e1b08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:40:17.417055 16290 consensus_queue.cc:1035] T 02c60ec285f84d08a38724bcc9add308 P f7d3440928d640e6ad9f508d12cadd33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be01008d6c3647208fe2f0acebe5742d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:40:17.425572 16297 mvcc.cc:205] Tried to move back new op lower bound from 7298580551328940032 to 7298580551234850816. Current Snapshot: MvccSnapshot[applied={T|T < 7298580551328940032}]
I20260619 14:40:17.433990 15806 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53702:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:40:17.434154 15806 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53702:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:40:17.436466 15806 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88: Sending DeleteTablet for 3 replicas of tablet 02c60ec285f84d08a38724bcc9add308
I20260619 14:40:17.437179 15931 tablet_service.cc:1514] Processing DeleteTablet for tablet 02c60ec285f84d08a38724bcc9add308 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:40:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35004
I20260619 14:40:17.437285 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 15868
I20260619 14:40:17.437443 16065 tablet_service.cc:1514] Processing DeleteTablet for tablet 02c60ec285f84d08a38724bcc9add308 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:40:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44202
I20260619 14:40:17.437829 16199 tablet_service.cc:1514] Processing DeleteTablet for tablet 02c60ec285f84d08a38724bcc9add308 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:40:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40938
I20260619 14:40:17.438266 16306 tablet_replica.cc:330] T 02c60ec285f84d08a38724bcc9add308 P be01008d6c3647208fe2f0acebe5742d: stopping tablet replica
I20260619 14:40:17.438585 16306 raft_consensus.cc:2229] T 02c60ec285f84d08a38724bcc9add308 P be01008d6c3647208fe2f0acebe5742d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:40:17.438898 16306 raft_consensus.cc:2258] T 02c60ec285f84d08a38724bcc9add308 P be01008d6c3647208fe2f0acebe5742d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:40:17.439273 16305 tablet_replica.cc:330] T 02c60ec285f84d08a38724bcc9add308 P f7d3440928d640e6ad9f508d12cadd33: stopping tablet replica
I20260619 14:40:17.439610 16305 raft_consensus.cc:2229] T 02c60ec285f84d08a38724bcc9add308 P f7d3440928d640e6ad9f508d12cadd33 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:40:17.440037 16305 raft_consensus.cc:2258] T 02c60ec285f84d08a38724bcc9add308 P f7d3440928d640e6ad9f508d12cadd33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:40:17.440510 16306 ts_tablet_manager.cc:1905] T 02c60ec285f84d08a38724bcc9add308 P be01008d6c3647208fe2f0acebe5742d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:40:17.441509 16305 ts_tablet_manager.cc:1905] T 02c60ec285f84d08a38724bcc9add308 P f7d3440928d640e6ad9f508d12cadd33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:40:17.443264 16306 ts_tablet_manager.cc:1918] T 02c60ec285f84d08a38724bcc9add308 P be01008d6c3647208fe2f0acebe5742d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:40:17.443369 16306 log.cc:1196] T 02c60ec285f84d08a38724bcc9add308 P be01008d6c3647208fe2f0acebe5742d: Deleting WAL directory at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-2/wal/wals/02c60ec285f84d08a38724bcc9add308
I20260619 14:40:17.443702 16306 ts_tablet_manager.cc:1939] T 02c60ec285f84d08a38724bcc9add308 P be01008d6c3647208fe2f0acebe5742d: Deleting consensus metadata
I20260619 14:40:17.444311 16305 ts_tablet_manager.cc:1918] T 02c60ec285f84d08a38724bcc9add308 P f7d3440928d640e6ad9f508d12cadd33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:40:17.444461 16305 log.cc:1196] T 02c60ec285f84d08a38724bcc9add308 P f7d3440928d640e6ad9f508d12cadd33: Deleting WAL directory at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-1/wal/wals/02c60ec285f84d08a38724bcc9add308
W20260619 14:40:17.444962 15793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45973 (error 108)
I20260619 14:40:17.444983 15790 catalog_manager.cc:4841] TS be01008d6c3647208fe2f0acebe5742d (127.0.98.3:35985): tablet 02c60ec285f84d08a38724bcc9add308 (table test-table [id=16e84011baba433593e7a61a0e2ae8fa]) successfully deleted
I20260619 14:40:17.445206 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 16002
I20260619 14:40:17.447160 16279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:17.446863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45973 (local address 127.0.0.1:35012)
0619 14:40:17.447013 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:17.447081 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260619 14:40:17.447345 15793 catalog_manager.cc:4584] TS f68c94798df842509a153535526e1b08 (127.0.98.1:45973): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 02c60ec285f84d08a38724bcc9add308: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45973: connect: Connection refused (error 111)
I20260619 14:40:17.450634 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 16136
I20260619 14:40:17.450915 16279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:17.450639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42139 (local address 127.0.0.1:44220)
0619 14:40:17.450785 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:17.450835 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:42139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 14:40:17.455483 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 15774
I20260619 14:40:37.461395 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42853
--webserver_interface=127.0.98.62
--webserver_port=33523
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42853
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:40:37.567884 16308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:40:37.568161 16308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:40:37.568215 16308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:40:37.572293 16308 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 14:40:37.572371 16308 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 14:40:37.572393 16308 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 14:40:37.572410 16308 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:40:37.572433 16308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:40:37.572458 16308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:40:37.572487 16308 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:40:37.572510 16308 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:40:37.577847 16308 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=33523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:40:37.578902 16308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:40:37.579892 16308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:40:37.592562 16315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:40:37.592576 16314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:40:37.592576 16317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:40:37.592638 16308 server_base.cc:1034] running on GCE node
I20260619 14:40:37.593490 16308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:40:37.594671 16308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:40:37.595870 16308 hybrid_clock.cc:648] HybridClock initialized: now 1781880037595844 us; error 35 us; skew 500 ppm
Jun 19 14:40:37 dist-test-slave-32hj krb5kdc[15761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:40:37.603400 16308 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:40:37.604398 16308 webserver.cc:460] Webserver started at http://127.0.98.62:33523/ using document root <none> and password file <none>
I20260619 14:40:37.605013 16308 fs_manager.cc:359] Metadata directory not provided
I20260619 14:40:37.605094 16308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:40:37.609267 16308 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:40:37.611596 16324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:40:37.612970 16308 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:40:37.613083 16308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/master-0/wal
uuid: "10cf4324bd7d41a39669534d9a7b4e88"
format_stamp: "Formatted at 2026-06-19 14:40:16 on dist-test-slave-32hj"
server_key: "67e38875f0ce57641f1f3b9bc52e7dc5"
server_key_iv: "1a6b7241774355dca89ba4ccad018798"
server_key_version: "encryptionkey@0"
I20260619 14:40:37.613682 16308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:40:37.648908 16308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:40:37.651921 16308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:40:37.652079 16308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:40:37.658946 16308 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42853
I20260619 14:40:37.659253 16376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42853 every 8 connection(s)
I20260619 14:40:37.660045 16308 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:40:37.664402 16377 sys_catalog.cc:263] Verifying existing consensus state
I20260619 14:40:37.666172 16377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88: Bootstrap starting.
I20260619 14:40:37.668236 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 16308
I20260619 14:40:37.668654 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45973
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:42853
--webserver_port=35437
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:40:37.687379 16377 log.cc:824] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88: Log is configured to *not* fsync() on all Append() calls
I20260619 14:40:37.698529 16377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:40:37.698952 16377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88: Bootstrap complete.
I20260619 14:40:37.704229 16377 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10cf4324bd7d41a39669534d9a7b4e88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42853 } }
I20260619 14:40:37.704751 16377 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10cf4324bd7d41a39669534d9a7b4e88, State: Initialized, Role: FOLLOWER
I20260619 14:40:37.705308 16377 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10cf4324bd7d41a39669534d9a7b4e88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42853 } }
I20260619 14:40:37.705451 16377 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:40:37.705520 16377 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:40:37.705611 16377 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88 [term 1 FOLLOWER]: Advancing to term 2
I20260619 14:40:37.708007 16377 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10cf4324bd7d41a39669534d9a7b4e88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42853 } }
I20260619 14:40:37.708369 16377 leader_election.cc:302] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10cf4324bd7d41a39669534d9a7b4e88; no voters:
I20260619 14:40:37.708752 16377 leader_election.cc:288] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 14:40:37.708922 16381 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 14:40:37.709438 16381 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88 [term 2 LEADER]: Becoming Leader. State: Replica: 10cf4324bd7d41a39669534d9a7b4e88, State: Running, Role: LEADER
I20260619 14:40:37.709826 16381 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10cf4324bd7d41a39669534d9a7b4e88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42853 } }
I20260619 14:40:37.710264 16377 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:40:37.711303 16383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "10cf4324bd7d41a39669534d9a7b4e88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10cf4324bd7d41a39669534d9a7b4e88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42853 } } }
I20260619 14:40:37.711289 16382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10cf4324bd7d41a39669534d9a7b4e88. Latest consensus state: current_term: 2 leader_uuid: "10cf4324bd7d41a39669534d9a7b4e88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10cf4324bd7d41a39669534d9a7b4e88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42853 } } }
I20260619 14:40:37.711431 16382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88 [sys.catalog]: This master's current role is: LEADER
I20260619 14:40:37.711431 16383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88 [sys.catalog]: This master's current role is: LEADER
I20260619 14:40:37.715617 16396 catalog_manager.cc:1259] Loaded cluster ID: 305d5b121e2d40a0a884286bc601db9b
I20260619 14:40:37.715706 16396 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88: loading cluster ID for follower catalog manager: success
I20260619 14:40:37.718382 16396 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88: acquiring CA information for follower catalog manager: success
I20260619 14:40:37.720036 16396 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 14:40:37.720182 16390 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:40:37.723127 16390 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:40:37.723655 16390 catalog_manager.cc:1259] Loaded cluster ID: 305d5b121e2d40a0a884286bc601db9b
I20260619 14:40:37.723714 16390 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:40:37.724431 16390 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:40:37.724972 16390 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88: Loaded TSK: 0
I20260619 14:40:37.743803 16390 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88: Generated new TSK 1
I20260619 14:40:37.744102 16390 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 14:40:37.795011 16379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:40:37.795293 16379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:40:37.795362 16379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:40:37.799633 16379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:40:37.799696 16379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:40:37.799779 16379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:40:37.805174 16379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45973
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35437
--tserver_master_addrs=127.0.98.62:42853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:40:37.806169 16379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:40:37.807147 16379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:40:37.816711 16405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:40:37.816707 16404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:40:37.817124 16407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:40:37.817106 16379 server_base.cc:1034] running on GCE node
I20260619 14:40:37.817582 16379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:40:37.818243 16379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:40:37.819413 16379 hybrid_clock.cc:648] HybridClock initialized: now 1781880037819399 us; error 26 us; skew 500 ppm
Jun 19 14:40:37 dist-test-slave-32hj krb5kdc[15761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:40:37.822207 16379 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:40:37.823120 16379 webserver.cc:460] Webserver started at http://127.0.98.1:35437/ using document root <none> and password file <none>
I20260619 14:40:37.823662 16379 fs_manager.cc:359] Metadata directory not provided
I20260619 14:40:37.823740 16379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:40:37.827634 16379 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:40:37.829857 16414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:40:37.831244 16379 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:40:37.831355 16379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-0/wal
uuid: "f68c94798df842509a153535526e1b08"
format_stamp: "Formatted at 2026-06-19 14:40:16 on dist-test-slave-32hj"
server_key: "f34f2e2b1c390ecaf1cf2f124f41a3b3"
server_key_iv: "a80b9e446beec75f72bd2f793c196c6b"
server_key_version: "encryptionkey@0"
I20260619 14:40:37.831882 16379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:40:37.855018 16379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:40:37.858238 16379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:40:37.858418 16379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:40:37.859020 16379 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:40:37.860073 16421 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 14:40:37.864110 16379 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 14:40:37.864193 16379 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260619 14:40:37.864269 16379 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 14:40:37.866014 16379 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 14:40:37.866065 16379 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.001s
I20260619 14:40:37.866319 16421 tablet_bootstrap.cc:492] T 02c60ec285f84d08a38724bcc9add308 P f68c94798df842509a153535526e1b08: Bootstrap starting.
I20260619 14:40:37.875686 16379 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45973
I20260619 14:40:37.875771 16528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45973 every 8 connection(s)
I20260619 14:40:37.876749 16379 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:40:37.885365 16421 log.cc:824] T 02c60ec285f84d08a38724bcc9add308 P f68c94798df842509a153535526e1b08: Log is configured to *not* fsync() on all Append() calls
I20260619 14:40:37.885955 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 16379
I20260619 14:40:37.886417 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:42139
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:42853
--webserver_port=38117
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:40:37 dist-test-slave-32hj krb5kdc[15761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:40:37.893610 16421 tablet_bootstrap.cc:492] T 02c60ec285f84d08a38724bcc9add308 P f68c94798df842509a153535526e1b08: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:40:37.894233 16421 tablet_bootstrap.cc:492] T 02c60ec285f84d08a38724bcc9add308 P f68c94798df842509a153535526e1b08: Bootstrap complete.
I20260619 14:40:37.894989 16532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:37.878712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42853 (local address 127.0.98.1:54713)
0619 14:40:37.879346 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:37.879401 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 14:40:37.880227 (+ 826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:40:37.881605 (+ 1378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:40:37.881621 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:40:37.881980 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:40:37.882467 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:40:37.882480 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:40:37.884453 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:40:37.884456 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:40:37.885397 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:40:37.885407 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:40:37.886247 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:40:37.886251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:40:37.886439 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:40:37.887047 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:40:37.887063 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:40:37.888679 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:40:37.891210 (+ 2531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:40:37.891217 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:40:37.891231 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:40:37.891690 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:40:37.892098 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:40:37.892101 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:40:37.892103 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:40:37.892303 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:40:37.893016 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:40:37.893024 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:40:37.893363 (+ 339us) client_negotiation.cc:769] Sending connection context
0619 14:40:37.893574 (+ 211us) client_negotiation.cc:240] Negotiation successful
0619 14:40:37.893940 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":138,"threads_started":1}
I20260619 14:40:37.895303 16531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:37.878796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54713 (local address 127.0.98.62:42853)
0619 14:40:37.879217 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0619 14:40:37.879241 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 14:40:37.879524 (+ 283us) server_negotiation.cc:407] Connection header received
0619 14:40:37.880700 (+ 1176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:40:37.880744 (+ 44us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:40:37.881161 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:40:37.881550 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:40:37.882633 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:40:37.884302 (+ 1669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:40:37.885520 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:40:37.886101 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:40:37.886258 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:40:37.888925 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:40:37.888932 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:40:37.888950 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:40:37.888983 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:40:37.890965 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:40:37.891828 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:40:37.891834 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:40:37.891840 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:40:37.891996 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:40:37.892403 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:40:37.892406 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:40:37.892408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:40:37.892895 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:40:37.893028 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 14:40:37.893911 (+ 883us) server_negotiation.cc:299] Negotiation successful
0619 14:40:37.894245 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":138,"threads_started":1}
I20260619 14:40:37.895558 16421 ts_tablet_manager.cc:1397] T 02c60ec285f84d08a38724bcc9add308 P f68c94798df842509a153535526e1b08: Time spent bootstrapping tablet: real 0.029s user 0.027s sys 0.000s
I20260619 14:40:37.897970 16529 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42853
I20260619 14:40:37.898202 16529 heartbeater.cc:464] Registering TS with master...
I20260619 14:40:37.898201 16421 raft_consensus.cc:348] T 02c60ec285f84d08a38724bcc9add308 P f68c94798df842509a153535526e1b08 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68c94798df842509a153535526e1b08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45973 } } peers { permanent_uuid: "be01008d6c3647208fe2f0acebe5742d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35985 } } peers { permanent_uuid: "f7d3440928d640e6ad9f508d12cadd33" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42139 } }
I20260619 14:40:37.898718 16421 raft_consensus.cc:729] T 02c60ec285f84d08a38724bcc9add308 P f68c94798df842509a153535526e1b08 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f68c94798df842509a153535526e1b08, State: Initialized, Role: FOLLOWER
I20260619 14:40:37.898769 16529 heartbeater.cc:511] Master 127.0.98.62:42853 requested a full tablet report, sending...
I20260619 14:40:37.899187 16421 consensus_queue.cc:260] T 02c60ec285f84d08a38724bcc9add308 P f68c94798df842509a153535526e1b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f68c94798df842509a153535526e1b08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45973 } } peers { permanent_uuid: "be01008d6c3647208fe2f0acebe5742d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35985 } } peers { permanent_uuid: "f7d3440928d640e6ad9f508d12cadd33" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42139 } }
I20260619 14:40:37.900224 16421 ts_tablet_manager.cc:1428] T 02c60ec285f84d08a38724bcc9add308 P f68c94798df842509a153535526e1b08: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:40:37.902204 16341 ts_manager.cc:194] Registered new tserver with Master: f68c94798df842509a153535526e1b08 (127.0.98.1:45973)
I20260619 14:40:37.906335 16341 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54713
I20260619 14:40:37.909348 16529 heartbeater.cc:503] Master 127.0.98.62:42853 was elected leader, sending a full tablet report...
Jun 19 14:40:37 dist-test-slave-32hj krb5kdc[15761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 14:40:37.915972 16537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:37.905704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48792 (local address 127.0.98.1:45973)
0619 14:40:37.906400 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0619 14:40:37.906416 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 14:40:37.906441 (+ 25us) server_negotiation.cc:407] Connection header received
0619 14:40:37.906586 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:40:37.906590 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:40:37.906758 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:40:37.906888 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:40:37.907583 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:40:37.908601 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:40:37.909857 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:40:37.910249 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:40:37.910346 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:40:37.912797 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:40:37.912808 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:40:37.912821 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:40:37.912843 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:40:37.914286 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:40:37.914765 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:40:37.914769 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:40:37.914771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:40:37.914841 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:40:37.915086 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:40:37.915088 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:40:37.915089 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:40:37.915306 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:40:37.915398 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 14:40:37.915726 (+ 328us) server_negotiation.cc:299] Negotiation successful
0619 14:40:37.915850 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":100,"threads_started":1}
I20260619 14:40:37.916011 16536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:37.905588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45973 (local address 127.0.0.1:48792)
0619 14:40:37.906196 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:37.906246 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 14:40:37.906429 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:40:37.906907 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:40:37.906913 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:40:37.907344 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:40:37.907450 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:40:37.907458 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:40:37.908760 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:40:37.908763 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:40:37.909738 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:40:37.909750 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:40:37.910348 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:40:37.910365 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:40:37.910477 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:40:37.911063 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:40:37.911077 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:40:37.912587 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:40:37.914390 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:40:37.914398 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:40:37.914404 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:40:37.914673 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:40:37.914917 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:40:37.914921 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:40:37.914922 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:40:37.915015 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:40:37.915400 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:40:37.915405 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:40:37.915527 (+ 122us) client_negotiation.cc:769] Sending connection context
0619 14:40:37.915666 (+ 139us) client_negotiation.cc:240] Negotiation successful
0619 14:40:37.915849 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":116,"threads_started":1}
I20260619 14:40:37.917019 16463 tablet_service.cc:1514] Processing DeleteTablet for tablet 02c60ec285f84d08a38724bcc9add308 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:40:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48792
I20260619 14:40:37.918040 16538 tablet_replica.cc:330] stopping tablet replica
I20260619 14:40:37.918349 16538 raft_consensus.cc:2229] T 02c60ec285f84d08a38724bcc9add308 P f68c94798df842509a153535526e1b08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:40:37.918578 16538 raft_consensus.cc:2258] T 02c60ec285f84d08a38724bcc9add308 P f68c94798df842509a153535526e1b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:40:37.919471 16538 ts_tablet_manager.cc:1905] T 02c60ec285f84d08a38724bcc9add308 P f68c94798df842509a153535526e1b08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:40:37.923161 16538 ts_tablet_manager.cc:1918] T 02c60ec285f84d08a38724bcc9add308 P f68c94798df842509a153535526e1b08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:40:37.923287 16538 log.cc:1196] T 02c60ec285f84d08a38724bcc9add308 P f68c94798df842509a153535526e1b08: Deleting WAL directory at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-0/wal/wals/02c60ec285f84d08a38724bcc9add308
I20260619 14:40:37.923514 16538 ts_tablet_manager.cc:1939] T 02c60ec285f84d08a38724bcc9add308 P f68c94798df842509a153535526e1b08: Deleting consensus metadata
I20260619 14:40:37.925174 16328 catalog_manager.cc:4841] TS f68c94798df842509a153535526e1b08 (127.0.98.1:45973): tablet 02c60ec285f84d08a38724bcc9add308 (table test-table [id=16e84011baba433593e7a61a0e2ae8fa]) successfully deleted
W20260619 14:40:38.002017 16534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:40:38.002254 16534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:40:38.002296 16534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:40:38.006247 16534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:40:38.006304 16534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:40:38.006361 16534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:40:38.011644 16534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:42139
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=38117
--tserver_master_addrs=127.0.98.62:42853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:40:38.012674 16534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:40:38.013654 16534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:40:38.023097 16543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:40:38.023092 16544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:40:38.023097 16546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:40:38.023190 16534 server_base.cc:1034] running on GCE node
I20260619 14:40:38.023723 16534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:40:38.024331 16534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:40:38.025483 16534 hybrid_clock.cc:648] HybridClock initialized: now 1781880038025463 us; error 29 us; skew 500 ppm
Jun 19 14:40:38 dist-test-slave-32hj krb5kdc[15761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880038, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:40:38.028026 16534 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:40:38.028937 16534 webserver.cc:460] Webserver started at http://127.0.98.2:38117/ using document root <none> and password file <none>
I20260619 14:40:38.029454 16534 fs_manager.cc:359] Metadata directory not provided
I20260619 14:40:38.029531 16534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:40:38.033293 16534 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260619 14:40:38.035323 16553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:40:38.036577 16534 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:40:38.036693 16534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-1/wal
uuid: "f7d3440928d640e6ad9f508d12cadd33"
format_stamp: "Formatted at 2026-06-19 14:40:17 on dist-test-slave-32hj"
server_key: "ac943e5cab4391b4dc461e33a05a02ce"
server_key_iv: "e2ec0d1056d386cfa8c0ba2c14146d7c"
server_key_version: "encryptionkey@0"
I20260619 14:40:38.037257 16534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:40:38.094652 16534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:40:38.097708 16534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:40:38.097889 16534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:40:38.098496 16534 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:40:38.099540 16560 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260619 14:40:38.103272 16560 ts_tablet_manager.cc:1852] T 02c60ec285f84d08a38724bcc9add308 P f7d3440928d640e6ad9f508d12cadd33: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260619 14:40:38.103703 16560 ts_tablet_manager.cc:1905] T 02c60ec285f84d08a38724bcc9add308 P f7d3440928d640e6ad9f508d12cadd33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:40:38.107612 16560 ts_tablet_manager.cc:1918] T 02c60ec285f84d08a38724bcc9add308 P f7d3440928d640e6ad9f508d12cadd33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:40:38.107758 16560 ts_tablet_manager.cc:1939] T 02c60ec285f84d08a38724bcc9add308 P f7d3440928d640e6ad9f508d12cadd33: Deleting consensus metadata
I20260619 14:40:38.108098 16534 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260619 14:40:38.108160 16534 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.000s
I20260619 14:40:38.108232 16534 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:40:38.108258 16534 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:40:38.117166 16534 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42139
I20260619 14:40:38.117230 16667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42139 every 8 connection(s)
I20260619 14:40:38.118079 16534 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:40:38.123907 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 16534
I20260619 14:40:38.124307 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:35985
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:42853
--webserver_port=39331
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:40:38 dist-test-slave-32hj krb5kdc[15761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880038, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:40:38.131614 16531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:38.119878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36039 (local address 127.0.98.62:42853)
0619 14:40:38.120012 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 14:40:38.120016 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:40:38.120894 (+ 878us) server_negotiation.cc:407] Connection header received
0619 14:40:38.121824 (+ 930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:40:38.121827 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:40:38.121892 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:40:38.121966 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:40:38.123155 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:40:38.123738 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:40:38.125083 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:40:38.125450 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:40:38.125565 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:40:38.128167 (+ 2602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:40:38.128170 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:40:38.128172 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:40:38.128200 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:40:38.129677 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:40:38.130153 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:40:38.130156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:40:38.130157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:40:38.130231 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:40:38.130538 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:40:38.130542 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:40:38.130543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:40:38.130721 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:40:38.130803 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 14:40:38.131317 (+ 514us) server_negotiation.cc:299] Negotiation successful
0619 14:40:38.131450 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 14:40:38.132424 16670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:38.120252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42853 (local address 127.0.98.2:36039)
0619 14:40:38.120703 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:38.120777 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0619 14:40:38.121619 (+ 842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:40:38.122149 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:40:38.122157 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:40:38.122516 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:40:38.123006 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:40:38.123021 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:40:38.123877 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:40:38.123880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:40:38.124962 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:40:38.124973 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:40:38.125643 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:40:38.125647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:40:38.125865 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:40:38.126496 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:40:38.126518 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:40:38.128001 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:40:38.129784 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:40:38.129790 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:40:38.129814 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:40:38.130061 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:40:38.130311 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:40:38.130314 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:40:38.130316 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:40:38.130454 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:40:38.130824 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:40:38.130830 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:40:38.131083 (+ 253us) client_negotiation.cc:769] Sending connection context
0619 14:40:38.131252 (+ 169us) client_negotiation.cc:240] Negotiation successful
0619 14:40:38.131582 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":102,"threads_started":1}
I20260619 14:40:38.133720 16668 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42853
I20260619 14:40:38.134020 16668 heartbeater.cc:464] Registering TS with master...
I20260619 14:40:38.134526 16668 heartbeater.cc:511] Master 127.0.98.62:42853 requested a full tablet report, sending...
I20260619 14:40:38.135592 16341 ts_manager.cc:194] Registered new tserver with Master: f7d3440928d640e6ad9f508d12cadd33 (127.0.98.2:42139)
I20260619 14:40:38.136217 16341 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36039
W20260619 14:40:38.231647 16671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:40:38.231894 16671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:40:38.231950 16671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:40:38.236024 16671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:40:38.236083 16671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:40:38.236160 16671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:40:38.241302 16671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:35985
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=39331
--tserver_master_addrs=127.0.98.62:42853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:40:38.242236 16671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:40:38.243194 16671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:40:38.252344 16676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:40:38.252370 16679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:40:38.252506 16671 server_base.cc:1034] running on GCE node
W20260619 14:40:38.252341 16677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:40:38.252919 16671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:40:38.253543 16671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:40:38.254712 16671 hybrid_clock.cc:648] HybridClock initialized: now 1781880038254673 us; error 51 us; skew 500 ppm
Jun 19 14:40:38 dist-test-slave-32hj krb5kdc[15761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880038, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:40:38.257362 16671 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:40:38.258203 16671 webserver.cc:460] Webserver started at http://127.0.98.3:39331/ using document root <none> and password file <none>
I20260619 14:40:38.258716 16671 fs_manager.cc:359] Metadata directory not provided
I20260619 14:40:38.258795 16671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:40:38.262490 16671 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:40:38.264379 16686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:40:38.265620 16671 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:40:38.265723 16671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-2/wal
uuid: "be01008d6c3647208fe2f0acebe5742d"
format_stamp: "Formatted at 2026-06-19 14:40:17 on dist-test-slave-32hj"
server_key: "71125b9862263840417123ad68383cce"
server_key_iv: "297aad49cc22b37b4f186ebe45e21c31"
server_key_version: "encryptionkey@0"
I20260619 14:40:38.266219 16671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:40:38.274673 16671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:40:38.277534 16671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:40:38.277696 16671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:40:38.278206 16671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:40:38.278939 16671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:40:38.278981 16671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:40:38.279031 16671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:40:38.279057 16671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:40:38.286969 16671 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35985
I20260619 14:40:38.287014 16799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35985 every 8 connection(s)
I20260619 14:40:38.287861 16671 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:40:38.290815 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 16671
Jun 19 14:40:38 dist-test-slave-32hj krb5kdc[15761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880038, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:40:38.299081 16804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:38.291433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37748 (local address 127.0.98.62:42853)
0619 14:40:38.291683 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0619 14:40:38.291687 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:40:38.291760 (+ 73us) server_negotiation.cc:407] Connection header received
0619 14:40:38.291880 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:40:38.291883 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:40:38.291929 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:40:38.292012 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:40:38.292708 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:40:38.293379 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:40:38.294274 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:40:38.294646 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:40:38.294746 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:40:38.295998 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:40:38.296001 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:40:38.296003 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:40:38.296024 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:40:38.297489 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:40:38.297848 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:40:38.297851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:40:38.297853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:40:38.297964 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:40:38.298207 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:40:38.298210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:40:38.298212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:40:38.298419 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:40:38.298626 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0619 14:40:38.298755 (+ 129us) server_negotiation.cc:299] Negotiation successful
0619 14:40:38.298901 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":83,"threads_started":1}
I20260619 14:40:38.301285 16531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:38.289576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35009 (local address 127.0.98.62:42853)
0619 14:40:38.289694 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 14:40:38.289698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:40:38.290652 (+ 954us) server_negotiation.cc:407] Connection header received
0619 14:40:38.291686 (+ 1034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:40:38.291690 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:40:38.291744 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:40:38.291820 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:40:38.293080 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:40:38.293660 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:40:38.294991 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:40:38.295235 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:40:38.295377 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:40:38.297822 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:40:38.297825 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:40:38.297828 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:40:38.297852 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:40:38.299320 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:40:38.299836 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:40:38.299839 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:40:38.299841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:40:38.299915 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:40:38.300207 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:40:38.300211 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:40:38.300212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:40:38.300397 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:40:38.300480 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 14:40:38.301039 (+ 559us) server_negotiation.cc:299] Negotiation successful
0619 14:40:38.301146 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 14:40:38.302037 16802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:38.289916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42853 (local address 127.0.98.3:35009)
0619 14:40:38.290494 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:38.290533 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 14:40:38.291470 (+ 937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:40:38.292018 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:40:38.292026 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:40:38.292383 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:40:38.292934 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:40:38.292953 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:40:38.293766 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:40:38.293769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:40:38.294876 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:40:38.294884 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:40:38.295410 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:40:38.295414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:40:38.295549 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:40:38.296009 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:40:38.296024 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:40:38.297687 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:40:38.299427 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:40:38.299433 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:40:38.299447 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:40:38.299739 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:40:38.299993 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:40:38.299995 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:40:38.299997 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:40:38.300118 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:40:38.300492 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:40:38.300498 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:40:38.300825 (+ 327us) client_negotiation.cc:769] Sending connection context
0619 14:40:38.300986 (+ 161us) client_negotiation.cc:240] Negotiation successful
0619 14:40:38.301274 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":101,"threads_started":1}
I20260619 14:40:38.302964 16800 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42853
I20260619 14:40:38.303164 16800 heartbeater.cc:464] Registering TS with master...
I20260619 14:40:38.303596 16800 heartbeater.cc:511] Master 127.0.98.62:42853 requested a full tablet report, sending...
I20260619 14:40:38.304437 16341 ts_manager.cc:194] Registered new tserver with Master: be01008d6c3647208fe2f0acebe5742d (127.0.98.3:35985)
I20260619 14:40:38.304991 16341 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35009
I20260619 14:40:38.312317 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:40:38.315856 16531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:38.312887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37764 (local address 127.0.98.62:42853)
0619 14:40:38.313010 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 14:40:38.313015 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:40:38.313196 (+ 181us) server_negotiation.cc:407] Connection header received
0619 14:40:38.313318 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:40:38.313320 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:40:38.313415 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:40:38.313495 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:40:38.313649 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:40:38.314130 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:40:38.314906 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:40:38.315109 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:40:38.315182 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:40:38.315435 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:40:38.315565 (+ 130us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0619 14:40:38.315761 (+ 196us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37764: token expired
Metrics: {"server-negotiator.queue_time_us":44}
W20260619 14:40:38.315925 16531 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37764: token expired
W20260619 14:40:38.316305 16269 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:42853)
I20260619 14:40:38.322819 16531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:38.316590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37766 (local address 127.0.98.62:42853)
0619 14:40:38.316703 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 14:40:38.316706 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:40:38.316762 (+ 56us) server_negotiation.cc:407] Connection header received
0619 14:40:38.316799 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:40:38.316802 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:40:38.316836 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:40:38.316897 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:40:38.317494 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:40:38.318015 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:40:38.318813 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:40:38.319016 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:40:38.319092 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:40:38.320209 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:40:38.320212 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:40:38.320214 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:40:38.320235 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:40:38.321553 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:40:38.321886 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:40:38.321888 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:40:38.321889 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:40:38.321958 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:40:38.322146 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:40:38.322148 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:40:38.322149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:40:38.322295 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:40:38.322387 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 14:40:38.322618 (+ 231us) server_negotiation.cc:299] Negotiation successful
0619 14:40:38.322703 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 14:40:38.325039 16341 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37766:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:40:38.328120 16341 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:40:38.336285 16463 tablet_service.cc:1467] Processing CreateTablet for tablet 3d47521a15834a15bbf853badfe1cb11 (DEFAULT_TABLE table=test-table [id=db3fb6e82e2241c1ae8c5c29ad04eb9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:40:38.336977 16463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d47521a15834a15bbf853badfe1cb11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:40:38.339730 16421 tablet_bootstrap.cc:492] T 3d47521a15834a15bbf853badfe1cb11 P f68c94798df842509a153535526e1b08: Bootstrap starting.
I20260619 14:40:38.340525 16536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:38.334894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42139 (local address 127.0.0.1:52036)
0619 14:40:38.335000 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:38.335015 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:40:38.335107 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:40:38.336275 (+ 1168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:40:38.336278 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:40:38.336310 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:40:38.336422 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:40:38.336428 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:40:38.338016 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:40:38.338020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:40:38.339121 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:40:38.339129 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:40:38.340090 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:40:38.340095 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:40:38.340196 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:40:38.340264 (+ 68us) client_negotiation.cc:769] Sending connection context
0619 14:40:38.340318 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 14:40:38.340398 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":3}
I20260619 14:40:38.340622 16421 tablet_bootstrap.cc:654] T 3d47521a15834a15bbf853badfe1cb11 P f68c94798df842509a153535526e1b08: Neither blocks nor log segments found. Creating new log.
I20260619 14:40:38.340705 16809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:38.335132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35985 (local address 127.0.0.1:40054)
0619 14:40:38.335420 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:38.335432 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:40:38.335516 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:40:38.336237 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:40:38.336240 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:40:38.336258 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:40:38.336422 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:40:38.336430 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:40:38.337775 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:40:38.337778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:40:38.339044 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:40:38.339054 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:40:38.340283 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:40:38.340287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:40:38.340361 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:40:38.340379 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 14:40:38.340451 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 14:40:38.340567 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":51,"spinlock_wait_cycles":24448,"thread_start_us":117,"threads_started":1}
I20260619 14:40:38.340744 16811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:38.335426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40054 (local address 127.0.98.3:35985)
0619 14:40:38.335820 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0619 14:40:38.335832 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:40:38.335853 (+ 21us) server_negotiation.cc:407] Connection header received
0619 14:40:38.335927 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:40:38.335931 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:40:38.336099 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:40:38.336288 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:40:38.336543 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:40:38.337682 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:40:38.339185 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:40:38.339952 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:40:38.340043 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:40:38.340166 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 14:40:38.340498 (+ 332us) server_negotiation.cc:299] Negotiation successful
0619 14:40:38.340592 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":116,"threads_started":1}
I20260619 14:40:38.341470 16810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:38.335116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52036 (local address 127.0.98.2:42139)
0619 14:40:38.335820 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0619 14:40:38.335830 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:40:38.335852 (+ 22us) server_negotiation.cc:407] Connection header received
0619 14:40:38.335928 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:40:38.335933 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:40:38.336124 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:40:38.336605 (+ 481us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:40:38.336801 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:40:38.337888 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:40:38.339209 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:40:38.339960 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:40:38.341023 (+ 1063us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:40:38.341172 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 14:40:38.341221 (+ 49us) server_negotiation.cc:299] Negotiation successful
0619 14:40:38.341311 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":125,"threads_started":1}
I20260619 14:40:38.342818 16602 tablet_service.cc:1467] Processing CreateTablet for tablet 3d47521a15834a15bbf853badfe1cb11 (DEFAULT_TABLE table=test-table [id=db3fb6e82e2241c1ae8c5c29ad04eb9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:40:38.343874 16602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d47521a15834a15bbf853badfe1cb11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:40:38.343948 16734 tablet_service.cc:1467] Processing CreateTablet for tablet 3d47521a15834a15bbf853badfe1cb11 (DEFAULT_TABLE table=test-table [id=db3fb6e82e2241c1ae8c5c29ad04eb9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:40:38.344995 16734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d47521a15834a15bbf853badfe1cb11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:40:38.348383 16560 tablet_bootstrap.cc:492] T 3d47521a15834a15bbf853badfe1cb11 P f7d3440928d640e6ad9f508d12cadd33: Bootstrap starting.
I20260619 14:40:38.349261 16421 tablet_bootstrap.cc:492] T 3d47521a15834a15bbf853badfe1cb11 P f68c94798df842509a153535526e1b08: No bootstrap required, opened a new log
I20260619 14:40:38.349367 16421 ts_tablet_manager.cc:1397] T 3d47521a15834a15bbf853badfe1cb11 P f68c94798df842509a153535526e1b08: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260619 14:40:38.349831 16421 raft_consensus.cc:348] T 3d47521a15834a15bbf853badfe1cb11 P f68c94798df842509a153535526e1b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d3440928d640e6ad9f508d12cadd33" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42139 } } peers { permanent_uuid: "be01008d6c3647208fe2f0acebe5742d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35985 } } peers { permanent_uuid: "f68c94798df842509a153535526e1b08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45973 } }
I20260619 14:40:38.349956 16421 raft_consensus.cc:374] T 3d47521a15834a15bbf853badfe1cb11 P f68c94798df842509a153535526e1b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:40:38.349993 16421 raft_consensus.cc:729] T 3d47521a15834a15bbf853badfe1cb11 P f68c94798df842509a153535526e1b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f68c94798df842509a153535526e1b08, State: Initialized, Role: FOLLOWER
I20260619 14:40:38.350153 16421 consensus_queue.cc:260] T 3d47521a15834a15bbf853badfe1cb11 P f68c94798df842509a153535526e1b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d3440928d640e6ad9f508d12cadd33" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42139 } } peers { permanent_uuid: "be01008d6c3647208fe2f0acebe5742d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35985 } } peers { permanent_uuid: "f68c94798df842509a153535526e1b08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45973 } }
I20260619 14:40:38.350512 16421 ts_tablet_manager.cc:1428] T 3d47521a15834a15bbf853badfe1cb11 P f68c94798df842509a153535526e1b08: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:40:38.350700 16560 tablet_bootstrap.cc:654] T 3d47521a15834a15bbf853badfe1cb11 P f7d3440928d640e6ad9f508d12cadd33: Neither blocks nor log segments found. Creating new log.
I20260619 14:40:38.351472 16560 log.cc:824] T 3d47521a15834a15bbf853badfe1cb11 P f7d3440928d640e6ad9f508d12cadd33: Log is configured to *not* fsync() on all Append() calls
I20260619 14:40:38.353396 16814 tablet_bootstrap.cc:492] T 3d47521a15834a15bbf853badfe1cb11 P be01008d6c3647208fe2f0acebe5742d: Bootstrap starting.
I20260619 14:40:38.353852 16560 tablet_bootstrap.cc:492] T 3d47521a15834a15bbf853badfe1cb11 P f7d3440928d640e6ad9f508d12cadd33: No bootstrap required, opened a new log
I20260619 14:40:38.354027 16560 ts_tablet_manager.cc:1397] T 3d47521a15834a15bbf853badfe1cb11 P f7d3440928d640e6ad9f508d12cadd33: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 14:40:38.355849 16814 tablet_bootstrap.cc:654] T 3d47521a15834a15bbf853badfe1cb11 P be01008d6c3647208fe2f0acebe5742d: Neither blocks nor log segments found. Creating new log.
I20260619 14:40:38.357100 16814 log.cc:824] T 3d47521a15834a15bbf853badfe1cb11 P be01008d6c3647208fe2f0acebe5742d: Log is configured to *not* fsync() on all Append() calls
I20260619 14:40:38.357772 16560 raft_consensus.cc:348] T 3d47521a15834a15bbf853badfe1cb11 P f7d3440928d640e6ad9f508d12cadd33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d3440928d640e6ad9f508d12cadd33" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42139 } } peers { permanent_uuid: "be01008d6c3647208fe2f0acebe5742d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35985 } } peers { permanent_uuid: "f68c94798df842509a153535526e1b08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45973 } }
I20260619 14:40:38.358033 16560 raft_consensus.cc:374] T 3d47521a15834a15bbf853badfe1cb11 P f7d3440928d640e6ad9f508d12cadd33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:40:38.358095 16560 raft_consensus.cc:729] T 3d47521a15834a15bbf853badfe1cb11 P f7d3440928d640e6ad9f508d12cadd33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7d3440928d640e6ad9f508d12cadd33, State: Initialized, Role: FOLLOWER
I20260619 14:40:38.358537 16560 consensus_queue.cc:260] T 3d47521a15834a15bbf853badfe1cb11 P f7d3440928d640e6ad9f508d12cadd33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d3440928d640e6ad9f508d12cadd33" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42139 } } peers { permanent_uuid: "be01008d6c3647208fe2f0acebe5742d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35985 } } peers { permanent_uuid: "f68c94798df842509a153535526e1b08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45973 } }
I20260619 14:40:38.358990 16814 tablet_bootstrap.cc:492] T 3d47521a15834a15bbf853badfe1cb11 P be01008d6c3647208fe2f0acebe5742d: No bootstrap required, opened a new log
I20260619 14:40:38.359118 16668 heartbeater.cc:503] Master 127.0.98.62:42853 was elected leader, sending a full tablet report...
I20260619 14:40:38.359163 16814 ts_tablet_manager.cc:1397] T 3d47521a15834a15bbf853badfe1cb11 P be01008d6c3647208fe2f0acebe5742d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 14:40:38.359520 16560 ts_tablet_manager.cc:1428] T 3d47521a15834a15bbf853badfe1cb11 P f7d3440928d640e6ad9f508d12cadd33: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 14:40:38.362210 16814 raft_consensus.cc:348] T 3d47521a15834a15bbf853badfe1cb11 P be01008d6c3647208fe2f0acebe5742d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d3440928d640e6ad9f508d12cadd33" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42139 } } peers { permanent_uuid: "be01008d6c3647208fe2f0acebe5742d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35985 } } peers { permanent_uuid: "f68c94798df842509a153535526e1b08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45973 } }
I20260619 14:40:38.362430 16814 raft_consensus.cc:374] T 3d47521a15834a15bbf853badfe1cb11 P be01008d6c3647208fe2f0acebe5742d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:40:38.362478 16814 raft_consensus.cc:729] T 3d47521a15834a15bbf853badfe1cb11 P be01008d6c3647208fe2f0acebe5742d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be01008d6c3647208fe2f0acebe5742d, State: Initialized, Role: FOLLOWER
I20260619 14:40:38.362856 16814 consensus_queue.cc:260] T 3d47521a15834a15bbf853badfe1cb11 P be01008d6c3647208fe2f0acebe5742d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d3440928d640e6ad9f508d12cadd33" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42139 } } peers { permanent_uuid: "be01008d6c3647208fe2f0acebe5742d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35985 } } peers { permanent_uuid: "f68c94798df842509a153535526e1b08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45973 } }
I20260619 14:40:38.363394 16800 heartbeater.cc:503] Master 127.0.98.62:42853 was elected leader, sending a full tablet report...
I20260619 14:40:38.363656 16814 ts_tablet_manager.cc:1428] T 3d47521a15834a15bbf853badfe1cb11 P be01008d6c3647208fe2f0acebe5742d: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
W20260619 14:40:38.368898 16669 tablet.cc:2369] T 3d47521a15834a15bbf853badfe1cb11 P f7d3440928d640e6ad9f508d12cadd33: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:40:38.377909 16530 tablet.cc:2369] T 3d47521a15834a15bbf853badfe1cb11 P f68c94798df842509a153535526e1b08: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:40:38.538763 16801 tablet.cc:2369] T 3d47521a15834a15bbf853badfe1cb11 P be01008d6c3647208fe2f0acebe5742d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:40:38.619802 16817 raft_consensus.cc:482] T 3d47521a15834a15bbf853badfe1cb11 P be01008d6c3647208fe2f0acebe5742d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:40:38.620018 16817 raft_consensus.cc:504] T 3d47521a15834a15bbf853badfe1cb11 P be01008d6c3647208fe2f0acebe5742d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d3440928d640e6ad9f508d12cadd33" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42139 } } peers { permanent_uuid: "be01008d6c3647208fe2f0acebe5742d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35985 } } peers { permanent_uuid: "f68c94798df842509a153535526e1b08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45973 } }
I20260619 14:40:38.621173 16817 leader_election.cc:288] T 3d47521a15834a15bbf853badfe1cb11 P be01008d6c3647208fe2f0acebe5742d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f7d3440928d640e6ad9f508d12cadd33 (127.0.98.2:42139), f68c94798df842509a153535526e1b08 (127.0.98.1:45973)
I20260619 14:40:38.625763 16810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:38.621582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34495 (local address 127.0.98.2:42139)
0619 14:40:38.621705 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 14:40:38.621710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:40:38.622007 (+ 297us) server_negotiation.cc:407] Connection header received
0619 14:40:38.622098 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:40:38.622101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:40:38.622147 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:40:38.622202 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:40:38.622447 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:40:38.623148 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:40:38.624602 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:40:38.625219 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:40:38.625320 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:40:38.625386 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 14:40:38.625566 (+ 180us) server_negotiation.cc:299] Negotiation successful
0619 14:40:38.625631 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 14:40:38.625766 16818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:38.621532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50993 (local address 127.0.98.1:45973)
0619 14:40:38.621811 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0619 14:40:38.621814 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:40:38.621825 (+ 11us) server_negotiation.cc:407] Connection header received
0619 14:40:38.621870 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:40:38.621873 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:40:38.621922 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:40:38.622021 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:40:38.622278 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:40:38.623041 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:40:38.624602 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:40:38.625298 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:40:38.625386 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:40:38.625521 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 14:40:38.625565 (+ 44us) server_negotiation.cc:299] Negotiation successful
0619 14:40:38.625631 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":99,"threads_started":1}
I20260619 14:40:38.625801 16802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:38.621402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45973 (local address 127.0.98.3:50993)
0619 14:40:38.621525 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:38.621541 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:40:38.621660 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:40:38.622041 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:40:38.622044 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:40:38.622058 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:40:38.622172 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:40:38.622178 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:40:38.623143 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:40:38.623145 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:40:38.624489 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:40:38.624500 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:40:38.625393 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:40:38.625397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:40:38.625487 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:40:38.625503 (+ 16us) client_negotiation.cc:769] Sending connection context
0619 14:40:38.625553 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 14:40:38.625659 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":30}
I20260619 14:40:38.625828 16819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:38.621403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42139 (local address 127.0.98.3:34495)
0619 14:40:38.621907 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:38.621918 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:40:38.622016 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:40:38.622223 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:40:38.622226 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:40:38.622242 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:40:38.622337 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:40:38.622343 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:40:38.623228 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:40:38.623231 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:40:38.624489 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:40:38.624500 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:40:38.625331 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:40:38.625334 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:40:38.625445 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:40:38.625470 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 14:40:38.625546 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 14:40:38.625649 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"mutex_wait_us":299,"thread_start_us":106,"threads_started":1}
I20260619 14:40:38.626418 16483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d47521a15834a15bbf853badfe1cb11" candidate_uuid: "be01008d6c3647208fe2f0acebe5742d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f68c94798df842509a153535526e1b08" is_pre_election: true
I20260619 14:40:38.626502 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d47521a15834a15bbf853badfe1cb11" candidate_uuid: "be01008d6c3647208fe2f0acebe5742d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7d3440928d640e6ad9f508d12cadd33" is_pre_election: true
I20260619 14:40:38.626715 16483 raft_consensus.cc:2454] T 3d47521a15834a15bbf853badfe1cb11 P f68c94798df842509a153535526e1b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be01008d6c3647208fe2f0acebe5742d in term 0.
I20260619 14:40:38.626767 16622 raft_consensus.cc:2454] T 3d47521a15834a15bbf853badfe1cb11 P f7d3440928d640e6ad9f508d12cadd33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be01008d6c3647208fe2f0acebe5742d in term 0.
I20260619 14:40:38.627139 16690 leader_election.cc:302] T 3d47521a15834a15bbf853badfe1cb11 P be01008d6c3647208fe2f0acebe5742d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be01008d6c3647208fe2f0acebe5742d, f68c94798df842509a153535526e1b08; no voters:
I20260619 14:40:38.627369 16817 raft_consensus.cc:2781] T 3d47521a15834a15bbf853badfe1cb11 P be01008d6c3647208fe2f0acebe5742d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:40:38.627449 16817 raft_consensus.cc:482] T 3d47521a15834a15bbf853badfe1cb11 P be01008d6c3647208fe2f0acebe5742d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:40:38.627497 16817 raft_consensus.cc:3037] T 3d47521a15834a15bbf853badfe1cb11 P be01008d6c3647208fe2f0acebe5742d [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:40:38.628473 16817 raft_consensus.cc:504] T 3d47521a15834a15bbf853badfe1cb11 P be01008d6c3647208fe2f0acebe5742d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d3440928d640e6ad9f508d12cadd33" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42139 } } peers { permanent_uuid: "be01008d6c3647208fe2f0acebe5742d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35985 } } peers { permanent_uuid: "f68c94798df842509a153535526e1b08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45973 } }
I20260619 14:40:38.628877 16817 leader_election.cc:288] T 3d47521a15834a15bbf853badfe1cb11 P be01008d6c3647208fe2f0acebe5742d [CANDIDATE]: Term 1 election: Requested vote from peers f7d3440928d640e6ad9f508d12cadd33 (127.0.98.2:42139), f68c94798df842509a153535526e1b08 (127.0.98.1:45973)
I20260619 14:40:38.629418 16483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d47521a15834a15bbf853badfe1cb11" candidate_uuid: "be01008d6c3647208fe2f0acebe5742d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f68c94798df842509a153535526e1b08"
I20260619 14:40:38.629545 16483 raft_consensus.cc:3037] T 3d47521a15834a15bbf853badfe1cb11 P f68c94798df842509a153535526e1b08 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:40:38.630509 16483 raft_consensus.cc:2454] T 3d47521a15834a15bbf853badfe1cb11 P f68c94798df842509a153535526e1b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be01008d6c3647208fe2f0acebe5742d in term 1.
I20260619 14:40:38.630735 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d47521a15834a15bbf853badfe1cb11" candidate_uuid: "be01008d6c3647208fe2f0acebe5742d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7d3440928d640e6ad9f508d12cadd33"
I20260619 14:40:38.630784 16690 leader_election.cc:302] T 3d47521a15834a15bbf853badfe1cb11 P be01008d6c3647208fe2f0acebe5742d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be01008d6c3647208fe2f0acebe5742d, f68c94798df842509a153535526e1b08; no voters:
I20260619 14:40:38.630847 16622 raft_consensus.cc:3037] T 3d47521a15834a15bbf853badfe1cb11 P f7d3440928d640e6ad9f508d12cadd33 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:40:38.630932 16817 raft_consensus.cc:2781] T 3d47521a15834a15bbf853badfe1cb11 P be01008d6c3647208fe2f0acebe5742d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:40:38.631143 16817 raft_consensus.cc:686] T 3d47521a15834a15bbf853badfe1cb11 P be01008d6c3647208fe2f0acebe5742d [term 1 LEADER]: Becoming Leader. State: Replica: be01008d6c3647208fe2f0acebe5742d, State: Running, Role: LEADER
I20260619 14:40:38.631507 16817 consensus_queue.cc:237] T 3d47521a15834a15bbf853badfe1cb11 P be01008d6c3647208fe2f0acebe5742d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d3440928d640e6ad9f508d12cadd33" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42139 } } peers { permanent_uuid: "be01008d6c3647208fe2f0acebe5742d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35985 } } peers { permanent_uuid: "f68c94798df842509a153535526e1b08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45973 } }
I20260619 14:40:38.632041 16622 raft_consensus.cc:2454] T 3d47521a15834a15bbf853badfe1cb11 P f7d3440928d640e6ad9f508d12cadd33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be01008d6c3647208fe2f0acebe5742d in term 1.
I20260619 14:40:38.634685 16340 catalog_manager.cc:5495] T 3d47521a15834a15bbf853badfe1cb11 P be01008d6c3647208fe2f0acebe5742d reported cstate change: term changed from 0 to 1, leader changed from <none> to be01008d6c3647208fe2f0acebe5742d (127.0.98.3). New cstate: current_term: 1 leader_uuid: "be01008d6c3647208fe2f0acebe5742d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d3440928d640e6ad9f508d12cadd33" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be01008d6c3647208fe2f0acebe5742d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35985 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f68c94798df842509a153535526e1b08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45973 } health_report { overall_health: UNKNOWN } } }
I20260619 14:40:38.708600 16811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:38.705251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40064 (local address 127.0.98.3:35985)
0619 14:40:38.705387 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 14:40:38.705392 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:40:38.705406 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:40:38.705549 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:40:38.705551 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:40:38.705608 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:40:38.705715 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:40:38.705945 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:40:38.706438 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:40:38.707229 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:40:38.707439 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:40:38.707523 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:40:38.707809 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:40:38.708217 (+ 408us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:40:38.708325 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 14:40:38.708390 (+ 65us) server_negotiation.cc:299] Negotiation successful
0619 14:40:38.708450 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 14:40:38.714627 16622 raft_consensus.cc:1261] T 3d47521a15834a15bbf853badfe1cb11 P f7d3440928d640e6ad9f508d12cadd33 [term 1 FOLLOWER]: Refusing update from remote peer be01008d6c3647208fe2f0acebe5742d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:40:38.714627 16483 raft_consensus.cc:1261] T 3d47521a15834a15bbf853badfe1cb11 P f68c94798df842509a153535526e1b08 [term 1 FOLLOWER]: Refusing update from remote peer be01008d6c3647208fe2f0acebe5742d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:40:38.715260 16820 consensus_queue.cc:1035] T 3d47521a15834a15bbf853badfe1cb11 P be01008d6c3647208fe2f0acebe5742d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f68c94798df842509a153535526e1b08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:40:38.715440 16817 consensus_queue.cc:1035] T 3d47521a15834a15bbf853badfe1cb11 P be01008d6c3647208fe2f0acebe5742d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7d3440928d640e6ad9f508d12cadd33" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:40:38.722555 16824 mvcc.cc:205] Tried to move back new op lower bound from 7298580638568058880 to 7298580638237659136. Current Snapshot: MvccSnapshot[applied={T|T < 7298580638568058880}]
I20260619 14:40:38.723690 16825 mvcc.cc:205] Tried to move back new op lower bound from 7298580638568058880 to 7298580638237659136. Current Snapshot: MvccSnapshot[applied={T|T < 7298580638568058880}]
I20260619 14:40:38.729688 16340 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:40:38.729833 16340 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:40:38.732025 16340 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88: Sending DeleteTablet for 3 replicas of tablet 3d47521a15834a15bbf853badfe1cb11
I20260619 14:40:38.732663 16463 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d47521a15834a15bbf853badfe1cb11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:40:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48792
I20260619 14:40:38.732797 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 16379
I20260619 14:40:38.732875 16602 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d47521a15834a15bbf853badfe1cb11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:40:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52036
I20260619 14:40:38.733603 16734 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d47521a15834a15bbf853badfe1cb11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:40:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40054
I20260619 14:40:38.734072 16833 tablet_replica.cc:330] T 3d47521a15834a15bbf853badfe1cb11 P f7d3440928d640e6ad9f508d12cadd33: stopping tablet replica
I20260619 14:40:38.734341 16833 raft_consensus.cc:2229] T 3d47521a15834a15bbf853badfe1cb11 P f7d3440928d640e6ad9f508d12cadd33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:40:38.734620 16833 raft_consensus.cc:2258] T 3d47521a15834a15bbf853badfe1cb11 P f7d3440928d640e6ad9f508d12cadd33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:40:38.735071 16834 tablet_replica.cc:330] T 3d47521a15834a15bbf853badfe1cb11 P be01008d6c3647208fe2f0acebe5742d: stopping tablet replica
I20260619 14:40:38.735409 16834 raft_consensus.cc:2229] T 3d47521a15834a15bbf853badfe1cb11 P be01008d6c3647208fe2f0acebe5742d [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:40:38.735807 16834 raft_consensus.cc:2258] T 3d47521a15834a15bbf853badfe1cb11 P be01008d6c3647208fe2f0acebe5742d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:40:38.736346 16833 ts_tablet_manager.cc:1905] T 3d47521a15834a15bbf853badfe1cb11 P f7d3440928d640e6ad9f508d12cadd33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:40:38.737237 16834 ts_tablet_manager.cc:1905] T 3d47521a15834a15bbf853badfe1cb11 P be01008d6c3647208fe2f0acebe5742d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:40:38.738971 16833 ts_tablet_manager.cc:1918] T 3d47521a15834a15bbf853badfe1cb11 P f7d3440928d640e6ad9f508d12cadd33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260619 14:40:38.739346 16328 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45973 (error 108)
I20260619 14:40:38.739161 16833 log.cc:1196] T 3d47521a15834a15bbf853badfe1cb11 P f7d3440928d640e6ad9f508d12cadd33: Deleting WAL directory at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-1/wal/wals/3d47521a15834a15bbf853badfe1cb11
I20260619 14:40:38.739847 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 16534
I20260619 14:40:38.739864 16834 ts_tablet_manager.cc:1918] T 3d47521a15834a15bbf853badfe1cb11 P be01008d6c3647208fe2f0acebe5742d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:40:38.739935 16834 log.cc:1196] T 3d47521a15834a15bbf853badfe1cb11 P be01008d6c3647208fe2f0acebe5742d: Deleting WAL directory at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-2/wal/wals/3d47521a15834a15bbf853badfe1cb11
I20260619 14:40:38.740585 16834 ts_tablet_manager.cc:1939] T 3d47521a15834a15bbf853badfe1cb11 P be01008d6c3647208fe2f0acebe5742d: Deleting consensus metadata
I20260619 14:40:38.741720 16809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:38.741454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45973 (local address 127.0.0.1:48808)
0619 14:40:38.741565 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:38.741635 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260619 14:40:38.741873 16328 catalog_manager.cc:4584] TS f68c94798df842509a153535526e1b08 (127.0.98.1:45973): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3d47521a15834a15bbf853badfe1cb11: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45973: connect: Connection refused (error 111)
I20260619 14:40:38.742050 16325 catalog_manager.cc:4841] TS be01008d6c3647208fe2f0acebe5742d (127.0.98.3:35985): tablet 3d47521a15834a15bbf853badfe1cb11 (table test-table [id=db3fb6e82e2241c1ae8c5c29ad04eb9c]) successfully deleted
I20260619 14:40:38.747202 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 16671
I20260619 14:40:38.747632 16809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:38.747363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42139 (local address 127.0.0.1:52040)
0619 14:40:38.747516 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:38.747560 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:42139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260619 14:40:38.752019 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 16308
I20260619 14:40:58.759306 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42853
--webserver_interface=127.0.98.62
--webserver_port=33523
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42853
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:40:58.869732 16836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:40:58.870029 16836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:40:58.870069 16836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:40:58.874138 16836 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 14:40:58.874217 16836 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 14:40:58.874233 16836 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 14:40:58.874245 16836 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:40:58.874262 16836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:40:58.874279 16836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:40:58.874297 16836 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:40:58.874325 16836 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:40:58.879843 16836 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=33523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:40:58.880987 16836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:40:58.882030 16836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:40:58.891480 16842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:40:58.891453 16841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:40:58.891434 16844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:40:58.891471 16836 server_base.cc:1034] running on GCE node
I20260619 14:40:58.892318 16836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:40:58.893662 16836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:40:58.894836 16836 hybrid_clock.cc:648] HybridClock initialized: now 1781880058894819 us; error 34 us; skew 500 ppm
Jun 19 14:40:58 dist-test-slave-32hj krb5kdc[15761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880058, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:40:58.902963 16836 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:40:58.904037 16836 webserver.cc:460] Webserver started at http://127.0.98.62:33523/ using document root <none> and password file <none>
I20260619 14:40:58.904624 16836 fs_manager.cc:359] Metadata directory not provided
I20260619 14:40:58.904695 16836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:40:58.908880 16836 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:40:58.911197 16851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:40:58.912487 16836 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:40:58.912607 16836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/master-0/wal
uuid: "10cf4324bd7d41a39669534d9a7b4e88"
format_stamp: "Formatted at 2026-06-19 14:40:16 on dist-test-slave-32hj"
server_key: "67e38875f0ce57641f1f3b9bc52e7dc5"
server_key_iv: "1a6b7241774355dca89ba4ccad018798"
server_key_version: "encryptionkey@0"
I20260619 14:40:58.913203 16836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:40:58.935400 16836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:40:58.939083 16836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:40:58.939298 16836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:40:58.946421 16836 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42853
I20260619 14:40:58.946678 16903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42853 every 8 connection(s)
I20260619 14:40:58.947479 16836 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:40:58.951833 16904 sys_catalog.cc:263] Verifying existing consensus state
I20260619 14:40:58.953614 16904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88: Bootstrap starting.
I20260619 14:40:58.956184 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 16836
I20260619 14:40:58.956671 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45973
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:42853
--webserver_port=35437
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:40:58.972162 16904 log.cc:824] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88: Log is configured to *not* fsync() on all Append() calls
I20260619 14:40:58.983758 16904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:40:58.984148 16904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88: Bootstrap complete.
I20260619 14:40:58.987998 16904 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10cf4324bd7d41a39669534d9a7b4e88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42853 } }
I20260619 14:40:58.988421 16904 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10cf4324bd7d41a39669534d9a7b4e88, State: Initialized, Role: FOLLOWER
I20260619 14:40:58.988932 16904 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10cf4324bd7d41a39669534d9a7b4e88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42853 } }
I20260619 14:40:58.989059 16904 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:40:58.989126 16904 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:40:58.989220 16904 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88 [term 2 FOLLOWER]: Advancing to term 3
I20260619 14:40:58.991210 16904 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10cf4324bd7d41a39669534d9a7b4e88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42853 } }
I20260619 14:40:58.991529 16904 leader_election.cc:302] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10cf4324bd7d41a39669534d9a7b4e88; no voters:
I20260619 14:40:58.991858 16904 leader_election.cc:288] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 14:40:58.991988 16908 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 14:40:58.992241 16908 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88 [term 3 LEADER]: Becoming Leader. State: Replica: 10cf4324bd7d41a39669534d9a7b4e88, State: Running, Role: LEADER
I20260619 14:40:58.992590 16908 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10cf4324bd7d41a39669534d9a7b4e88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42853 } }
I20260619 14:40:58.993153 16904 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:40:58.994485 16909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "10cf4324bd7d41a39669534d9a7b4e88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10cf4324bd7d41a39669534d9a7b4e88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42853 } } }
I20260619 14:40:58.994608 16909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88 [sys.catalog]: This master's current role is: LEADER
I20260619 14:40:58.995033 16910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10cf4324bd7d41a39669534d9a7b4e88. Latest consensus state: current_term: 3 leader_uuid: "10cf4324bd7d41a39669534d9a7b4e88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10cf4324bd7d41a39669534d9a7b4e88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42853 } } }
I20260619 14:40:58.995081 16918 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:40:58.995205 16910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88 [sys.catalog]: This master's current role is: LEADER
I20260619 14:40:59.001650 16918 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:40:59.002422 16918 catalog_manager.cc:1259] Loaded cluster ID: 305d5b121e2d40a0a884286bc601db9b
I20260619 14:40:59.002529 16918 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:40:59.004983 16918 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:40:59.005682 16918 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88: Loaded TSK: 0
I20260619 14:40:59.005764 16918 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 10cf4324bd7d41a39669534d9a7b4e88: Loaded TSK: 1
F20260619 14:40:59.006520 16918 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f3405c1e9ad google::LogMessage::Fail() at ??:0
@ 0x7f3405c2188a google::LogMessage::SendToLog() at ??:0
@ 0x7f3405c1e4de google::LogMessage::Flush() at ??:0
@ 0x7f3405c1fa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f34088e218d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f34088daf0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f34105837bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f341055d99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f341055e178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f341059fdf8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f34105be82b std::function<>::operator()() at ??:0
@ 0x7f341055db4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f341055ef3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f341055b579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f341059f640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f3407e703d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f3407e70c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f3407e723f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f3407e61512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f34094a56db start_thread at ??:0
@ 0x7f340471c61f clone at ??:0
W20260619 14:40:59.097689 16906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:40:59.097921 16906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:40:59.097965 16906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:40:59.102231 16906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:40:59.102294 16906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:40:59.102380 16906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:40:59.107870 16906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45973
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35437
--tserver_master_addrs=127.0.98.62:42853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:40:59.109048 16906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:40:59.110191 16906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:40:59.120183 16954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:40:59.120203 16952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:40:59.120203 16951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:40:59.120534 16906 server_base.cc:1034] running on GCE node
I20260619 14:40:59.121233 16906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:40:59.122054 16906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:40:59.123237 16906 hybrid_clock.cc:648] HybridClock initialized: now 1781880059123219 us; error 36 us; skew 500 ppm
Jun 19 14:40:59 dist-test-slave-32hj krb5kdc[15761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880059, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:40:59.125999 16906 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:40:59.126953 16906 webserver.cc:460] Webserver started at http://127.0.98.1:35437/ using document root <none> and password file <none>
I20260619 14:40:59.127508 16906 fs_manager.cc:359] Metadata directory not provided
I20260619 14:40:59.127573 16906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:40:59.131595 16906 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:40:59.133886 16961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:40:59.135254 16906 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:40:59.135383 16906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-0/wal
uuid: "f68c94798df842509a153535526e1b08"
format_stamp: "Formatted at 2026-06-19 14:40:16 on dist-test-slave-32hj"
server_key: "f34f2e2b1c390ecaf1cf2f124f41a3b3"
server_key_iv: "a80b9e446beec75f72bd2f793c196c6b"
server_key_version: "encryptionkey@0"
I20260619 14:40:59.135938 16906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:40:59.151916 16906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:40:59.155344 16906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:40:59.155537 16906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:40:59.156150 16906 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:40:59.157337 16968 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 14:40:59.161427 16906 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 14:40:59.161487 16906 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260619 14:40:59.161536 16906 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 14:40:59.163216 16906 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 14:40:59.163269 16906 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260619 14:40:59.163616 16968 tablet_bootstrap.cc:492] T 3d47521a15834a15bbf853badfe1cb11 P f68c94798df842509a153535526e1b08: Bootstrap starting.
I20260619 14:40:59.172979 16906 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45973
I20260619 14:40:59.173265 17075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45973 every 8 connection(s)
I20260619 14:40:59.174015 16906 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:40:59.183682 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 16906
I20260619 14:40:59.184095 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:42139
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:42853
--webserver_port=38117
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:40:59.185606 16968 log.cc:824] T 3d47521a15834a15bbf853badfe1cb11 P f68c94798df842509a153535526e1b08: Log is configured to *not* fsync() on all Append() calls
I20260619 14:40:59.195261 16968 tablet_bootstrap.cc:492] T 3d47521a15834a15bbf853badfe1cb11 P f68c94798df842509a153535526e1b08: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:40:59.195672 16968 tablet_bootstrap.cc:492] T 3d47521a15834a15bbf853badfe1cb11 P f68c94798df842509a153535526e1b08: Bootstrap complete.
I20260619 14:40:59.196957 16968 ts_tablet_manager.cc:1397] T 3d47521a15834a15bbf853badfe1cb11 P f68c94798df842509a153535526e1b08: Time spent bootstrapping tablet: real 0.034s user 0.022s sys 0.011s
I20260619 14:40:59.200299 16968 raft_consensus.cc:348] T 3d47521a15834a15bbf853badfe1cb11 P f68c94798df842509a153535526e1b08 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d3440928d640e6ad9f508d12cadd33" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42139 } } peers { permanent_uuid: "be01008d6c3647208fe2f0acebe5742d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35985 } } peers { permanent_uuid: "f68c94798df842509a153535526e1b08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45973 } }
I20260619 14:40:59.200934 16968 raft_consensus.cc:729] T 3d47521a15834a15bbf853badfe1cb11 P f68c94798df842509a153535526e1b08 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f68c94798df842509a153535526e1b08, State: Initialized, Role: FOLLOWER
I20260619 14:40:59.201543 16968 consensus_queue.cc:260] T 3d47521a15834a15bbf853badfe1cb11 P f68c94798df842509a153535526e1b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d3440928d640e6ad9f508d12cadd33" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42139 } } peers { permanent_uuid: "be01008d6c3647208fe2f0acebe5742d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35985 } } peers { permanent_uuid: "f68c94798df842509a153535526e1b08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45973 } }
I20260619 14:40:59.202755 16968 ts_tablet_manager.cc:1428] T 3d47521a15834a15bbf853badfe1cb11 P f68c94798df842509a153535526e1b08: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260619 14:40:59.241321 17078 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:40:59.175873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42853 (local address 127.0.98.1:51573)
0619 14:40:59.176333 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:59.176386 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 14:40:59.177208 (+ 822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:40:59.240300 (+ 63092us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42853: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":102,"threads_started":1}
W20260619 14:40:59.241822 17076 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42853 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:42853: Client connection negotiation failed: client connection to 127.0.98.62:42853: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 14:40:59.242559 17078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:59.242299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42853 (local address 127.0.98.1:47965)
0619 14:40:59.242426 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:59.242494 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 14:40:59.243196 17078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:59.242996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42853 (local address 127.0.98.1:50549)
0619 14:40:59.243092 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:59.243141 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260619 14:40:59.243386 17076 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 14:40:59.295657 17079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:40:59.295917 17079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:40:59.295958 17079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:40:59.299927 17079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:40:59.299988 17079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:40:59.300052 17079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:40:59.305137 17079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:42139
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=38117
--tserver_master_addrs=127.0.98.62:42853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:40:59.306135 17079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:40:59.307104 17079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:40:59.316182 17086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:40:59.316376 17079 server_base.cc:1034] running on GCE node
W20260619 14:40:59.316195 17087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:40:59.316186 17089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:40:59.316913 17079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:40:59.317515 17079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:40:59.318688 17079 hybrid_clock.cc:648] HybridClock initialized: now 1781880059318667 us; error 33 us; skew 500 ppm
Jun 19 14:40:59 dist-test-slave-32hj krb5kdc[15761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880059, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:40:59.321328 17079 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:40:59.322186 17079 webserver.cc:460] Webserver started at http://127.0.98.2:38117/ using document root <none> and password file <none>
I20260619 14:40:59.322713 17079 fs_manager.cc:359] Metadata directory not provided
I20260619 14:40:59.322779 17079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:40:59.326469 17079 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:40:59.328502 17096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:40:59.329962 17079 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:40:59.330093 17079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-1/wal
uuid: "f7d3440928d640e6ad9f508d12cadd33"
format_stamp: "Formatted at 2026-06-19 14:40:17 on dist-test-slave-32hj"
server_key: "ac943e5cab4391b4dc461e33a05a02ce"
server_key_iv: "e2ec0d1056d386cfa8c0ba2c14146d7c"
server_key_version: "encryptionkey@0"
I20260619 14:40:59.330631 17079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:40:59.347723 17079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:40:59.350731 17079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:40:59.350889 17079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:40:59.351447 17079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:40:59.352563 17103 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260619 14:40:59.356451 17103 ts_tablet_manager.cc:1852] T 3d47521a15834a15bbf853badfe1cb11 P f7d3440928d640e6ad9f508d12cadd33: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260619 14:40:59.356961 17103 ts_tablet_manager.cc:1905] T 3d47521a15834a15bbf853badfe1cb11 P f7d3440928d640e6ad9f508d12cadd33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:40:59.361037 17103 ts_tablet_manager.cc:1918] T 3d47521a15834a15bbf853badfe1cb11 P f7d3440928d640e6ad9f508d12cadd33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:40:59.361141 17103 log.cc:1196] T 3d47521a15834a15bbf853badfe1cb11 P f7d3440928d640e6ad9f508d12cadd33: Deleting WAL directory at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-1/wal/wals/3d47521a15834a15bbf853badfe1cb11
I20260619 14:40:59.361474 17103 ts_tablet_manager.cc:1939] T 3d47521a15834a15bbf853badfe1cb11 P f7d3440928d640e6ad9f508d12cadd33: Deleting consensus metadata
I20260619 14:40:59.361773 17079 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260619 14:40:59.361851 17079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.000s sys 0.000s
I20260619 14:40:59.361908 17079 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:40:59.361930 17079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:40:59.370453 17079 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42139
I20260619 14:40:59.370551 17210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42139 every 8 connection(s)
I20260619 14:40:59.371342 17079 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:40:59.374678 17213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:59.373417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42853 (local address 127.0.98.2:40915)
0619 14:40:59.373874 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:59.373993 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":112,"threads_started":1}
W20260619 14:40:59.375034 17211 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42853 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:42853: Client connection negotiation failed: client connection to 127.0.98.62:42853: connect: Connection refused (error 111)
I20260619 14:40:59.375517 17213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:59.375332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42853 (local address 127.0.98.2:37157)
0619 14:40:59.375424 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:59.375466 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260619 14:40:59.376050 17213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:59.375889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42853 (local address 127.0.98.2:54727)
0619 14:40:59.375969 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:59.376006 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260619 14:40:59.376169 17211 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 14:40:59.380846 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 17079
I20260619 14:40:59.381268 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:35985
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:42853
--webserver_port=39331
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:40:59.492786 17214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:40:59.493075 17214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:40:59.493120 17214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:40:59.497150 17214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:40:59.497210 17214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:40:59.497269 17214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:40:59.502521 17214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:35985
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=39331
--tserver_master_addrs=127.0.98.62:42853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17214
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:40:59.503559 17214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:40:59.504524 17214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:40:59.513891 17214 server_base.cc:1034] running on GCE node
W20260619 14:40:59.513778 17222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:40:59.513772 17219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:40:59.513770 17220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:40:59.514452 17214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:40:59.515079 17214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:40:59.516232 17214 hybrid_clock.cc:648] HybridClock initialized: now 1781880059516215 us; error 31 us; skew 500 ppm
Jun 19 14:40:59 dist-test-slave-32hj krb5kdc[15761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880059, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:40:59.518731 17214 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:40:59.519578 17214 webserver.cc:460] Webserver started at http://127.0.98.3:39331/ using document root <none> and password file <none>
I20260619 14:40:59.520094 17214 fs_manager.cc:359] Metadata directory not provided
I20260619 14:40:59.520157 17214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:40:59.523921 17214 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:40:59.525949 17229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:40:59.527141 17214 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:40:59.527241 17214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-2/wal
uuid: "be01008d6c3647208fe2f0acebe5742d"
format_stamp: "Formatted at 2026-06-19 14:40:17 on dist-test-slave-32hj"
server_key: "71125b9862263840417123ad68383cce"
server_key_iv: "297aad49cc22b37b4f186ebe45e21c31"
server_key_version: "encryptionkey@0"
I20260619 14:40:59.527746 17214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:40:59.566474 17214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:40:59.569453 17214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:40:59.569635 17214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:40:59.570218 17214 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:40:59.571027 17214 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:40:59.571074 17214 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:40:59.571127 17214 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:40:59.571153 17214 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:40:59.579514 17214 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35985
I20260619 14:40:59.579542 17342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35985 every 8 connection(s)
I20260619 14:40:59.580577 17214 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:40:59.584270 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:59.582766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42853 (local address 127.0.98.3:55123)
0619 14:40:59.583430 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:59.583540 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":109,"threads_started":1}
W20260619 14:40:59.584769 17343 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42853 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:42853: Client connection negotiation failed: client connection to 127.0.98.62:42853: connect: Connection refused (error 111)
I20260619 14:40:59.585354 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:59.585153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42853 (local address 127.0.98.3:54459)
0619 14:40:59.585251 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:59.585298 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260619 14:40:59.585892 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:40:59.585703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42853 (local address 127.0.98.3:53467)
0619 14:40:59.585786 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0619 14:40:59.585844 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260619 14:40:59.586041 17343 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 14:40:59.588016 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 17214
I20260619 14:41:00.244431 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:00.243968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42853 (local address 127.0.98.1:45855)
0619 14:41:00.244228 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:00.244305 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":77,"threads_started":1}
I20260619 14:41:00.376986 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:00.376626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42853 (local address 127.0.98.2:59759)
0619 14:41:00.376847 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:00.376901 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":90,"threads_started":1}
I20260619 14:41:00.586964 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:00.586538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42853 (local address 127.0.98.3:39113)
0619 14:41:00.586804 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:00.586863 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":69,"threads_started":1}
I20260619 14:41:01.044305 17350 raft_consensus.cc:482] T 3d47521a15834a15bbf853badfe1cb11 P f68c94798df842509a153535526e1b08 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:41:01.044488 17350 raft_consensus.cc:504] T 3d47521a15834a15bbf853badfe1cb11 P f68c94798df842509a153535526e1b08 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d3440928d640e6ad9f508d12cadd33" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42139 } } peers { permanent_uuid: "be01008d6c3647208fe2f0acebe5742d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35985 } } peers { permanent_uuid: "f68c94798df842509a153535526e1b08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45973 } }
I20260619 14:41:01.045593 17350 leader_election.cc:288] T 3d47521a15834a15bbf853badfe1cb11 P f68c94798df842509a153535526e1b08 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f7d3440928d640e6ad9f508d12cadd33 (127.0.98.2:42139), be01008d6c3647208fe2f0acebe5742d (127.0.98.3:35985)
Jun 19 14:41:01 dist-test-slave-32hj krb5kdc[15761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880059, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Jun 19 14:41:01 dist-test-slave-32hj krb5kdc[15761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880059, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260619 14:41:01.059803 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:01.045858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42139 (local address 127.0.98.1:51647)
0619 14:41:01.046118 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:01.046135 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:01.046248 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:01.048008 (+ 1760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:01.048013 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:01.048562 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:41:01.049083 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:01.049099 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:01.050325 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:01.050328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:01.051283 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:01.051292 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:01.052201 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:01.052205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:01.052424 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:01.053189 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:41:01.053206 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:41:01.055469 (+ 2263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:41:01.057623 (+ 2154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:41:01.057626 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:41:01.057628 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:41:01.057842 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:41:01.058138 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:41:01.058141 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:41:01.058143 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:41:01.058272 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:41:01.058849 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:41:01.058856 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:41:01.059100 (+ 244us) client_negotiation.cc:769] Sending connection context
0619 14:41:01.059263 (+ 163us) client_negotiation.cc:240] Negotiation successful
0619 14:41:01.059604 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":137,"spinlock_wait_cycles":19840,"thread_start_us":96,"threads_started":1}
I20260619 14:41:01.059818 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:01.045993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51647 (local address 127.0.98.2:42139)
0619 14:41:01.046463 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:01.046482 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:01.046527 (+ 45us) server_negotiation.cc:407] Connection header received
0619 14:41:01.047265 (+ 738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:01.047281 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:01.047663 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:01.047953 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:01.049239 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:01.050202 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:01.051405 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:01.052064 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:01.052767 (+ 703us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:01.055668 (+ 2901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:01.055674 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:01.055685 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:41:01.055717 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:01.057520 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:01.057939 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:01.057945 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:01.057950 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:01.058054 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:01.058355 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:01.058358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:01.058360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:01.058741 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:01.058860 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:01.059390 (+ 530us) server_negotiation.cc:299] Negotiation successful
0619 14:41:01.059632 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":93,"threads_started":1}
I20260619 14:41:01.059823 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:01.046386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56073 (local address 127.0.98.3:35985)
0619 14:41:01.046630 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:01.046648 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:01.046681 (+ 33us) server_negotiation.cc:407] Connection header received
0619 14:41:01.047329 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:01.047334 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:01.047648 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:01.047936 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:01.049265 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:01.050186 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:01.051443 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:01.052103 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:01.052261 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:01.055668 (+ 3407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:01.055674 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:01.055685 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:41:01.055718 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:01.057397 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:01.057918 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:01.057923 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:01.057927 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:01.058038 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:01.058350 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:01.058353 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:01.058355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:01.058750 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:01.058861 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:01.059389 (+ 528us) server_negotiation.cc:299] Negotiation successful
0619 14:41:01.059631 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":63,"threads_started":1}
I20260619 14:41:01.059839 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:01.046193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35985 (local address 127.0.98.1:56073)
0619 14:41:01.046455 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:01.046467 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:01.046552 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:01.048006 (+ 1454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:01.048012 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:01.048562 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:41:01.049083 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:01.049099 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:01.050323 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:01.050327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:01.051285 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:01.051317 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:01.052265 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:01.052277 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:01.052425 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:01.053150 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:41:01.053172 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:41:01.055470 (+ 2298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:41:01.057505 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:41:01.057512 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:41:01.057525 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:41:01.057812 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:41:01.058124 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:41:01.058127 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:41:01.058129 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:41:01.058267 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:41:01.058854 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:41:01.058857 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:41:01.059100 (+ 243us) client_negotiation.cc:769] Sending connection context
0619 14:41:01.059262 (+ 162us) client_negotiation.cc:240] Negotiation successful
0619 14:41:01.059677 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":293,"spinlock_wait_cycles":36736,"thread_start_us":67,"threads_started":1}
W20260619 14:41:01.062500 16962 leader_election.cc:341] T 3d47521a15834a15bbf853badfe1cb11 P f68c94798df842509a153535526e1b08 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f7d3440928d640e6ad9f508d12cadd33 (127.0.98.2:42139): Not found: Tablet not found: 3d47521a15834a15bbf853badfe1cb11
W20260619 14:41:01.062693 16962 leader_election.cc:341] T 3d47521a15834a15bbf853badfe1cb11 P f68c94798df842509a153535526e1b08 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer be01008d6c3647208fe2f0acebe5742d (127.0.98.3:35985): Not found: Tablet not found: 3d47521a15834a15bbf853badfe1cb11
I20260619 14:41:01.062738 16962 leader_election.cc:302] T 3d47521a15834a15bbf853badfe1cb11 P f68c94798df842509a153535526e1b08 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f68c94798df842509a153535526e1b08; no voters: be01008d6c3647208fe2f0acebe5742d, f7d3440928d640e6ad9f508d12cadd33
I20260619 14:41:01.062952 17350 raft_consensus.cc:2726] T 3d47521a15834a15bbf853badfe1cb11 P f68c94798df842509a153535526e1b08 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 14:41:01.245471 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:01.245164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42853 (local address 127.0.98.1:60107)
0619 14:41:01.245319 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:01.245386 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260619 14:41:01.378172 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:01.377742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42853 (local address 127.0.98.2:48563)
0619 14:41:01.377998 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:01.378064 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":77,"threads_started":1}
I20260619 14:41:01.588174 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:01.587754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42853 (local address 127.0.98.3:41885)
0619 14:41:01.588023 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:01.588084 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":74,"threads_started":1}
I20260619 14:41:02.246611 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:02.246178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42853 (local address 127.0.98.1:49973)
0619 14:41:02.246445 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:02.246512 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":79,"threads_started":1}
I20260619 14:41:02.379453 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:02.378987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42853 (local address 127.0.98.2:55715)
0619 14:41:02.379258 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:02.379323 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":90,"threads_started":1}
I20260619 14:41:02.589550 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:02.589061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42853 (local address 127.0.98.3:50959)
0619 14:41:02.589375 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:02.589440 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":93,"threads_started":1}
I20260619 14:41:02.717757 17360 raft_consensus.cc:482] T 3d47521a15834a15bbf853badfe1cb11 P f68c94798df842509a153535526e1b08 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:41:02.717897 17360 raft_consensus.cc:504] T 3d47521a15834a15bbf853badfe1cb11 P f68c94798df842509a153535526e1b08 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d3440928d640e6ad9f508d12cadd33" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42139 } } peers { permanent_uuid: "be01008d6c3647208fe2f0acebe5742d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35985 } } peers { permanent_uuid: "f68c94798df842509a153535526e1b08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45973 } }
I20260619 14:41:02.718287 17360 leader_election.cc:288] T 3d47521a15834a15bbf853badfe1cb11 P f68c94798df842509a153535526e1b08 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f7d3440928d640e6ad9f508d12cadd33 (127.0.98.2:42139), be01008d6c3647208fe2f0acebe5742d (127.0.98.3:35985)
W20260619 14:41:02.719063 16962 leader_election.cc:341] T 3d47521a15834a15bbf853badfe1cb11 P f68c94798df842509a153535526e1b08 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f7d3440928d640e6ad9f508d12cadd33 (127.0.98.2:42139): Not found: Tablet not found: 3d47521a15834a15bbf853badfe1cb11
W20260619 14:41:02.719199 16962 leader_election.cc:341] T 3d47521a15834a15bbf853badfe1cb11 P f68c94798df842509a153535526e1b08 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer be01008d6c3647208fe2f0acebe5742d (127.0.98.3:35985): Not found: Tablet not found: 3d47521a15834a15bbf853badfe1cb11
I20260619 14:41:02.719247 16962 leader_election.cc:302] T 3d47521a15834a15bbf853badfe1cb11 P f68c94798df842509a153535526e1b08 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f68c94798df842509a153535526e1b08; no voters: be01008d6c3647208fe2f0acebe5742d, f7d3440928d640e6ad9f508d12cadd33
I20260619 14:41:02.719421 17360 raft_consensus.cc:2726] T 3d47521a15834a15bbf853badfe1cb11 P f68c94798df842509a153535526e1b08 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 14:41:03.247646 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:03.247268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42853 (local address 127.0.98.1:53883)
0619 14:41:03.247499 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:03.247553 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":71,"threads_started":1}
I20260619 14:41:03.380642 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:03.380177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42853 (local address 127.0.98.2:53111)
0619 14:41:03.380443 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:03.380503 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":75,"threads_started":1}
I20260619 14:41:03.590689 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:03.590230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42853 (local address 127.0.98.3:47447)
0619 14:41:03.590541 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:03.590601 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":84,"threads_started":1}
I20260619 14:41:04.248977 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:04.248471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42853 (local address 127.0.98.1:45439)
0619 14:41:04.248819 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:04.248881 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":88,"threads_started":1}
I20260619 14:41:04.381997 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:04.381532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42853 (local address 127.0.98.2:47519)
0619 14:41:04.381805 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:04.381880 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":79,"threads_started":1}
I20260619 14:41:04.591852 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:04.591438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42853 (local address 127.0.98.3:44815)
0619 14:41:04.591706 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:04.591761 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":79,"threads_started":1}
I20260619 14:41:05.250023 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:05.249652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42853 (local address 127.0.98.1:33321)
0619 14:41:05.249878 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:05.249931 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":75,"threads_started":1}
I20260619 14:41:05.383134 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:05.382734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42853 (local address 127.0.98.2:56563)
0619 14:41:05.382961 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:05.383017 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":76,"threads_started":1}
I20260619 14:41:05.593014 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:05.592546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42853 (local address 127.0.98.3:50981)
0619 14:41:05.592841 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:05.592909 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":74,"threads_started":1}
I20260619 14:41:06.068445 17370 raft_consensus.cc:482] T 3d47521a15834a15bbf853badfe1cb11 P f68c94798df842509a153535526e1b08 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:41:06.068584 17370 raft_consensus.cc:504] T 3d47521a15834a15bbf853badfe1cb11 P f68c94798df842509a153535526e1b08 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d3440928d640e6ad9f508d12cadd33" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42139 } } peers { permanent_uuid: "be01008d6c3647208fe2f0acebe5742d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35985 } } peers { permanent_uuid: "f68c94798df842509a153535526e1b08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45973 } }
I20260619 14:41:06.069038 17370 leader_election.cc:288] T 3d47521a15834a15bbf853badfe1cb11 P f68c94798df842509a153535526e1b08 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f7d3440928d640e6ad9f508d12cadd33 (127.0.98.2:42139), be01008d6c3647208fe2f0acebe5742d (127.0.98.3:35985)
W20260619 14:41:06.069849 16962 leader_election.cc:341] T 3d47521a15834a15bbf853badfe1cb11 P f68c94798df842509a153535526e1b08 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f7d3440928d640e6ad9f508d12cadd33 (127.0.98.2:42139): Not found: Tablet not found: 3d47521a15834a15bbf853badfe1cb11
W20260619 14:41:06.069976 16962 leader_election.cc:341] T 3d47521a15834a15bbf853badfe1cb11 P f68c94798df842509a153535526e1b08 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer be01008d6c3647208fe2f0acebe5742d (127.0.98.3:35985): Not found: Tablet not found: 3d47521a15834a15bbf853badfe1cb11
I20260619 14:41:06.070026 16962 leader_election.cc:302] T 3d47521a15834a15bbf853badfe1cb11 P f68c94798df842509a153535526e1b08 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f68c94798df842509a153535526e1b08; no voters: be01008d6c3647208fe2f0acebe5742d, f7d3440928d640e6ad9f508d12cadd33
I20260619 14:41:06.070184 17370 raft_consensus.cc:2726] T 3d47521a15834a15bbf853badfe1cb11 P f68c94798df842509a153535526e1b08 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 14:41:06.251055 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:06.250657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42853 (local address 127.0.98.1:44579)
0619 14:41:06.250908 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:06.250961 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":72,"threads_started":1}
I20260619 14:41:06.384361 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:06.383887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42853 (local address 127.0.98.2:40933)
0619 14:41:06.384185 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:06.384250 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":74,"threads_started":1}
I20260619 14:41:06.594275 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:06.593760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42853 (local address 127.0.98.3:51551)
0619 14:41:06.594097 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:06.594163 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":73,"threads_started":1}
I20260619 14:41:07.252117 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:07.251696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42853 (local address 127.0.98.1:55043)
0619 14:41:07.251969 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:07.252025 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":73,"threads_started":1}
I20260619 14:41:07.385669 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:07.385202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42853 (local address 127.0.98.2:43691)
0619 14:41:07.385500 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:07.385567 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":74,"threads_started":1}
I20260619 14:41:07.595335 17376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:07.594961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42853 (local address 127.0.98.3:43739)
0619 14:41:07.595188 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:07.595245 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":68,"threads_started":1}
I20260619 14:41:07.897416 17377 raft_consensus.cc:482] T 3d47521a15834a15bbf853badfe1cb11 P f68c94798df842509a153535526e1b08 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:41:07.897584 17377 raft_consensus.cc:504] T 3d47521a15834a15bbf853badfe1cb11 P f68c94798df842509a153535526e1b08 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d3440928d640e6ad9f508d12cadd33" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42139 } } peers { permanent_uuid: "be01008d6c3647208fe2f0acebe5742d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35985 } } peers { permanent_uuid: "f68c94798df842509a153535526e1b08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45973 } }
I20260619 14:41:07.898017 17377 leader_election.cc:288] T 3d47521a15834a15bbf853badfe1cb11 P f68c94798df842509a153535526e1b08 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f7d3440928d640e6ad9f508d12cadd33 (127.0.98.2:42139), be01008d6c3647208fe2f0acebe5742d (127.0.98.3:35985)
W20260619 14:41:07.898784 16962 leader_election.cc:341] T 3d47521a15834a15bbf853badfe1cb11 P f68c94798df842509a153535526e1b08 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f7d3440928d640e6ad9f508d12cadd33 (127.0.98.2:42139): Not found: Tablet not found: 3d47521a15834a15bbf853badfe1cb11
W20260619 14:41:07.898908 16962 leader_election.cc:341] T 3d47521a15834a15bbf853badfe1cb11 P f68c94798df842509a153535526e1b08 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer be01008d6c3647208fe2f0acebe5742d (127.0.98.3:35985): Not found: Tablet not found: 3d47521a15834a15bbf853badfe1cb11
I20260619 14:41:07.898957 16962 leader_election.cc:302] T 3d47521a15834a15bbf853badfe1cb11 P f68c94798df842509a153535526e1b08 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f68c94798df842509a153535526e1b08; no voters: be01008d6c3647208fe2f0acebe5742d, f7d3440928d640e6ad9f508d12cadd33
I20260619 14:41:07.899082 17377 raft_consensus.cc:2726] T 3d47521a15834a15bbf853badfe1cb11 P f68c94798df842509a153535526e1b08 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 14:41:08.253353 17378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:08.252944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42853 (local address 127.0.98.1:46381)
0619 14:41:08.253201 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:08.253256 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":91,"threads_started":1}
I20260619 14:41:08.386729 17379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:08.386344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42853 (local address 127.0.98.2:34817)
0619 14:41:08.386581 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:08.386639 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":74,"threads_started":1}
I20260619 14:41:08.596385 17380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:08.596001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42853 (local address 127.0.98.3:44699)
0619 14:41:08.596242 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:08.596296 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":76,"threads_started":1}
I20260619 14:41:09.254734 17381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:09.254269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42853 (local address 127.0.98.1:49833)
0619 14:41:09.254575 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:09.254637 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":76,"threads_started":1}
I20260619 14:41:09.387957 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:09.387480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42853 (local address 127.0.98.2:49223)
0619 14:41:09.387774 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:09.387836 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":81,"threads_started":1}
I20260619 14:41:09.597573 17383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:09.597150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42853 (local address 127.0.98.3:32815)
0619 14:41:09.597407 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:09.597471 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":72,"threads_started":1}
I20260619 14:41:10.255954 17384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:10.255545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42853 (local address 127.0.98.1:58939)
0619 14:41:10.255791 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:10.255845 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":79,"threads_started":1}
I20260619 14:41:10.389137 17385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:10.388663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42853 (local address 127.0.98.2:52345)
0619 14:41:10.388952 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:10.389013 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":91,"threads_started":1}
I20260619 14:41:10.598752 17386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:10.598334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42853 (local address 127.0.98.3:40985)
0619 14:41:10.598596 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:10.598657 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":85,"threads_started":1}
I20260619 14:41:11.257251 17387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:11.256807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42853 (local address 127.0.98.1:34371)
0619 14:41:11.257091 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:11.257155 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":108,"threads_started":1}
I20260619 14:41:11.390309 17388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:11.389847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42853 (local address 127.0.98.2:58745)
0619 14:41:11.390138 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:11.390197 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":87,"threads_started":1}
I20260619 14:41:11.599825 17389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:11.599415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42853 (local address 127.0.98.3:40799)
0619 14:41:11.599675 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:11.599732 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":70,"threads_started":1}
I20260619 14:41:12.258392 17390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:12.257978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42853 (local address 127.0.98.1:46931)
0619 14:41:12.258228 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:12.258288 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":79,"threads_started":1}
I20260619 14:41:12.391475 17391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:12.391030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42853 (local address 127.0.98.2:60673)
0619 14:41:12.391307 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:12.391379 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":73,"threads_started":1}
I20260619 14:41:12.601068 17392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:12.600577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42853 (local address 127.0.98.3:55017)
0619 14:41:12.600894 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:12.600959 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":89,"threads_started":1}
I20260619 14:41:13.259672 17393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:13.259222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42853 (local address 127.0.98.1:52921)
0619 14:41:13.259510 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:13.259576 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":83,"threads_started":1}
I20260619 14:41:13.392864 17394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:13.392349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42853 (local address 127.0.98.2:53579)
0619 14:41:13.392651 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:13.392753 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":97,"threads_started":1}
I20260619 14:41:13.441704 17395 raft_consensus.cc:482] T 3d47521a15834a15bbf853badfe1cb11 P f68c94798df842509a153535526e1b08 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:41:13.441829 17395 raft_consensus.cc:504] T 3d47521a15834a15bbf853badfe1cb11 P f68c94798df842509a153535526e1b08 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d3440928d640e6ad9f508d12cadd33" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42139 } } peers { permanent_uuid: "be01008d6c3647208fe2f0acebe5742d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35985 } } peers { permanent_uuid: "f68c94798df842509a153535526e1b08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45973 } }
I20260619 14:41:13.442221 17395 leader_election.cc:288] T 3d47521a15834a15bbf853badfe1cb11 P f68c94798df842509a153535526e1b08 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f7d3440928d640e6ad9f508d12cadd33 (127.0.98.2:42139), be01008d6c3647208fe2f0acebe5742d (127.0.98.3:35985)
W20260619 14:41:13.442958 16962 leader_election.cc:341] T 3d47521a15834a15bbf853badfe1cb11 P f68c94798df842509a153535526e1b08 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer be01008d6c3647208fe2f0acebe5742d (127.0.98.3:35985): Not found: Tablet not found: 3d47521a15834a15bbf853badfe1cb11
W20260619 14:41:13.443085 16962 leader_election.cc:341] T 3d47521a15834a15bbf853badfe1cb11 P f68c94798df842509a153535526e1b08 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f7d3440928d640e6ad9f508d12cadd33 (127.0.98.2:42139): Not found: Tablet not found: 3d47521a15834a15bbf853badfe1cb11
I20260619 14:41:13.443122 16962 leader_election.cc:302] T 3d47521a15834a15bbf853badfe1cb11 P f68c94798df842509a153535526e1b08 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f68c94798df842509a153535526e1b08; no voters: be01008d6c3647208fe2f0acebe5742d, f7d3440928d640e6ad9f508d12cadd33
I20260619 14:41:13.443253 17395 raft_consensus.cc:2726] T 3d47521a15834a15bbf853badfe1cb11 P f68c94798df842509a153535526e1b08 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 14:41:13.602134 17396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:13.601730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42853 (local address 127.0.98.3:34223)
0619 14:41:13.601971 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:13.602040 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":89,"threads_started":1}
I20260619 14:41:14.261101 17397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:14.260647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42853 (local address 127.0.98.1:47397)
0619 14:41:14.260946 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:14.261005 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":103,"threads_started":1}
I20260619 14:41:14.394275 17398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:14.393740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42853 (local address 127.0.98.2:44833)
0619 14:41:14.394107 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:14.394173 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":90,"threads_started":1}
I20260619 14:41:14.595189 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 16906
I20260619 14:41:14.600188 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 17079
I20260619 14:41:14.603426 17399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:14.603017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42853 (local address 127.0.98.3:45359)
0619 14:41:14.603256 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:14.603335 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":65,"threads_started":1}
I20260619 14:41:14.605257 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 17214
2026-06-19T14:41:14Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59294 ms)
[----------] 30 tests from SecurityITest (236949 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:41:14 dist-test-slave-32hj krb5kdc[17402](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:41:14 dist-test-slave-32hj krb5kdc[17402](info): set up 2 sockets
Jun 19 14:41:14 dist-test-slave-32hj krb5kdc[17402](info): commencing operation
krb5kdc: starting...
W20260619 14:41:15.564371 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/krb5kdc/test-user.keytab.
Jun 19 14:41:15 dist-test-slave-32hj krb5kdc[17402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:41:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:41:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:41:15.682931 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42301
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42301
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:41:15.790383 17415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:41:15.790643 17415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:41:15.790697 17415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:41:15.794652 17415 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:41:15.794720 17415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:41:15.794747 17415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:41:15.794767 17415 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:41:15.794787 17415 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:41:15.800058 17415 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42301
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:41:15.801213 17415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:41:15.802219 17415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:41:15.810869 17420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:15.810923 17415 server_base.cc:1034] running on GCE node
W20260619 14:41:15.810851 17421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:15.810851 17423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:15.811661 17415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:41:15.812839 17415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:41:15.814002 17415 hybrid_clock.cc:648] HybridClock initialized: now 1781880075813986 us; error 30 us; skew 500 ppm
Jun 19 14:41:15 dist-test-slave-32hj krb5kdc[17402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:41:15.816553 17415 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:41:15.817421 17415 webserver.cc:460] Webserver started at http://127.0.98.62:41315/ using document root <none> and password file <none>
I20260619 14:41:15.817955 17415 fs_manager.cc:359] Metadata directory not provided
I20260619 14:41:15.818019 17415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:41:15.818203 17415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:41:15.820190 17415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/master-0/data/instance:
uuid: "79df4cbd968a412fac2675ea730ad180"
format_stamp: "Formatted at 2026-06-19 14:41:15 on dist-test-slave-32hj"
server_key: "ce490656888d4a73611a7bf3869ce2e2"
server_key_iv: "833a8f33277146aa03a09108c82ae259"
server_key_version: "encryptionkey@0"
I20260619 14:41:15.820704 17415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/master-0/wal/instance:
uuid: "79df4cbd968a412fac2675ea730ad180"
format_stamp: "Formatted at 2026-06-19 14:41:15 on dist-test-slave-32hj"
server_key: "ce490656888d4a73611a7bf3869ce2e2"
server_key_iv: "833a8f33277146aa03a09108c82ae259"
server_key_version: "encryptionkey@0"
I20260619 14:41:15.824163 17415 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:41:15.826429 17430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:15.827612 17415 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:41:15.827726 17415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/master-0/wal
uuid: "79df4cbd968a412fac2675ea730ad180"
format_stamp: "Formatted at 2026-06-19 14:41:15 on dist-test-slave-32hj"
server_key: "ce490656888d4a73611a7bf3869ce2e2"
server_key_iv: "833a8f33277146aa03a09108c82ae259"
server_key_version: "encryptionkey@0"
I20260619 14:41:15.827817 17415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:41:15.844164 17415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:41:15.846972 17415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:41:15.847149 17415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:41:15.853996 17415 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42301
I20260619 14:41:15.854010 17482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42301 every 8 connection(s)
I20260619 14:41:15.854928 17415 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:41:15.857676 17483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:41:15.858811 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 17415
I20260619 14:41:15.858920 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/master-0/wal/instance
I20260619 14:41:15.859200 392 external_mini_cluster.cc:1428] Setting key e4632c7ca2a760594b3051d9acb6c8c8
I20260619 14:41:15.864133 17483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79df4cbd968a412fac2675ea730ad180: Bootstrap starting.
I20260619 14:41:15.866695 17483 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79df4cbd968a412fac2675ea730ad180: Neither blocks nor log segments found. Creating new log.
I20260619 14:41:15.867452 17483 log.cc:824] T 00000000000000000000000000000000 P 79df4cbd968a412fac2675ea730ad180: Log is configured to *not* fsync() on all Append() calls
Jun 19 14:41:15 dist-test-slave-32hj krb5kdc[17402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:41:15.869592 17483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79df4cbd968a412fac2675ea730ad180: No bootstrap required, opened a new log
I20260619 14:41:15.873847 17486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:15.860549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60072 (local address 127.0.98.62:42301)
0619 14:41:15.860968 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:15.860987 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:15.861029 (+ 42us) server_negotiation.cc:407] Connection header received
0619 14:41:15.861810 (+ 781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:15.861833 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:15.862263 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:15.862633 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:15.863305 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:15.864698 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:15.865574 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:15.866245 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:15.866396 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:15.868860 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:15.868866 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:15.868883 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:41:15.868913 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:15.870774 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:15.871173 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:15.871179 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:15.871185 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:15.871319 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:15.871537 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:15.871541 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:15.871542 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:15.872093 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:15.872271 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:15.872472 (+ 201us) server_negotiation.cc:299] Negotiation successful
0619 14:41:15.872800 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":126,"threads_started":1}
I20260619 14:41:15.874002 17483 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79df4cbd968a412fac2675ea730ad180 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79df4cbd968a412fac2675ea730ad180" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42301 } }
I20260619 14:41:15.874269 17483 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79df4cbd968a412fac2675ea730ad180 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:41:15.874325 17483 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79df4cbd968a412fac2675ea730ad180 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79df4cbd968a412fac2675ea730ad180, State: Initialized, Role: FOLLOWER
I20260619 14:41:15.874790 17483 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79df4cbd968a412fac2675ea730ad180 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79df4cbd968a412fac2675ea730ad180" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42301 } }
I20260619 14:41:15.874953 17483 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79df4cbd968a412fac2675ea730ad180 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:41:15.875020 17483 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79df4cbd968a412fac2675ea730ad180 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:41:15.875113 17483 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79df4cbd968a412fac2675ea730ad180 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:41:15.876029 17483 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79df4cbd968a412fac2675ea730ad180 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79df4cbd968a412fac2675ea730ad180" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42301 } }
I20260619 14:41:15.876361 17483 leader_election.cc:302] T 00000000000000000000000000000000 P 79df4cbd968a412fac2675ea730ad180 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79df4cbd968a412fac2675ea730ad180; no voters:
I20260619 14:41:15.876624 17483 leader_election.cc:288] T 00000000000000000000000000000000 P 79df4cbd968a412fac2675ea730ad180 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:41:15.876781 17488 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79df4cbd968a412fac2675ea730ad180 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:41:15.876984 17488 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79df4cbd968a412fac2675ea730ad180 [term 1 LEADER]: Becoming Leader. State: Replica: 79df4cbd968a412fac2675ea730ad180, State: Running, Role: LEADER
I20260619 14:41:15.877296 17488 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79df4cbd968a412fac2675ea730ad180 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79df4cbd968a412fac2675ea730ad180" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42301 } }
I20260619 14:41:15.877767 17483 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79df4cbd968a412fac2675ea730ad180 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:41:15.878746 17490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79df4cbd968a412fac2675ea730ad180 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79df4cbd968a412fac2675ea730ad180. Latest consensus state: current_term: 1 leader_uuid: "79df4cbd968a412fac2675ea730ad180" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79df4cbd968a412fac2675ea730ad180" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42301 } } }
I20260619 14:41:15.878865 17490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79df4cbd968a412fac2675ea730ad180 [sys.catalog]: This master's current role is: LEADER
I20260619 14:41:15.878997 17489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79df4cbd968a412fac2675ea730ad180 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79df4cbd968a412fac2675ea730ad180" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79df4cbd968a412fac2675ea730ad180" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42301 } } }
I20260619 14:41:15.879144 17489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79df4cbd968a412fac2675ea730ad180 [sys.catalog]: This master's current role is: LEADER
W20260619 14:41:15.882310 17503 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 79df4cbd968a412fac2675ea730ad180: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:41:15.882380 17503 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:41:15.882433 17497 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:41:15.883090 17497 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:41:15.888402 17497 catalog_manager.cc:1347] Generated new cluster ID: a3128ab7773b46b799c37c7b4edd00df
I20260619 14:41:15.888479 17497 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:41:15.907107 17497 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:41:15.908406 17497 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:41:15.914880 17497 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79df4cbd968a412fac2675ea730ad180: Generated new TSK 0
I20260619 14:41:15.915676 17497 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 14:41:15.944162 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42301
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:41:16.052269 17509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:41:16.052546 17509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:41:16.052608 17509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:41:16.057097 17509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:41:16.057166 17509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:41:16.057260 17509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:41:16.062615 17509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:41:16.063686 17509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:41:16.064664 17509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:41:16.073805 17515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:16.073812 17514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:16.073849 17509 server_base.cc:1034] running on GCE node
W20260619 14:41:16.073812 17517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:16.074442 17509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:41:16.075017 17509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:41:16.076200 17509 hybrid_clock.cc:648] HybridClock initialized: now 1781880076076180 us; error 29 us; skew 500 ppm
Jun 19 14:41:16 dist-test-slave-32hj krb5kdc[17402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:41:16.079283 17509 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:41:16.080209 17509 webserver.cc:460] Webserver started at http://127.0.98.1:46257/ using document root <none> and password file <none>
I20260619 14:41:16.080787 17509 fs_manager.cc:359] Metadata directory not provided
I20260619 14:41:16.080852 17509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:41:16.081041 17509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:41:16.083014 17509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/ts-0/data/instance:
uuid: "97a837f775c5419690c4e49710fce43f"
format_stamp: "Formatted at 2026-06-19 14:41:16 on dist-test-slave-32hj"
server_key: "a89c922c1ce33ba41c3725bcc72e6e56"
server_key_iv: "f6235ef6da9717248d36b5cdfc0d76fb"
server_key_version: "encryptionkey@0"
I20260619 14:41:16.083532 17509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/ts-0/wal/instance:
uuid: "97a837f775c5419690c4e49710fce43f"
format_stamp: "Formatted at 2026-06-19 14:41:16 on dist-test-slave-32hj"
server_key: "a89c922c1ce33ba41c3725bcc72e6e56"
server_key_iv: "f6235ef6da9717248d36b5cdfc0d76fb"
server_key_version: "encryptionkey@0"
I20260619 14:41:16.087214 17509 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:41:16.089792 17524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:16.091010 17509 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:41:16.091132 17509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/ts-0/wal
uuid: "97a837f775c5419690c4e49710fce43f"
format_stamp: "Formatted at 2026-06-19 14:41:16 on dist-test-slave-32hj"
server_key: "a89c922c1ce33ba41c3725bcc72e6e56"
server_key_iv: "f6235ef6da9717248d36b5cdfc0d76fb"
server_key_version: "encryptionkey@0"
I20260619 14:41:16.091231 17509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:41:16.111971 17509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:41:16.114898 17509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:41:16.115069 17509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:41:16.115638 17509 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:41:16.116426 17509 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:41:16.116472 17509 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:16.116523 17509 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:41:16.116549 17509 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:16.124763 17509 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40773
I20260619 14:41:16.124799 17637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40773 every 8 connection(s)
I20260619 14:41:16.125617 17509 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:41:16.130503 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 17509
I20260619 14:41:16.130599 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:41:16.130869 392 external_mini_cluster.cc:1428] Setting key 82b6b80636c9118e361d0f96ed04447c
Jun 19 14:41:16 dist-test-slave-32hj krb5kdc[17402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:41:16.138655 17486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:16.127325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49111 (local address 127.0.98.62:42301)
0619 14:41:16.127440 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:16.127444 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:16.128236 (+ 792us) server_negotiation.cc:407] Connection header received
0619 14:41:16.129075 (+ 839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:16.129078 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:16.129126 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:16.129196 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:16.130308 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:16.130950 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:16.132341 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:16.132600 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:16.132687 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:16.135289 (+ 2602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:16.135293 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:16.135296 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:41:16.135322 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:16.136683 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:16.137183 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:16.137187 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:16.137189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:16.137269 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:16.137578 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:16.137581 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:16.137583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:16.137764 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:16.137849 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:16.138348 (+ 499us) server_negotiation.cc:299] Negotiation successful
0619 14:41:16.138486 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 14:41:16.139472 17640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:16.127630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42301 (local address 127.0.98.1:49111)
0619 14:41:16.128071 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:16.128119 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:16.128879 (+ 760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:16.129325 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:16.129331 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:16.129672 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:41:16.130166 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:16.130180 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:16.131060 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:16.131064 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:16.132208 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:16.132219 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:16.132775 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:16.132780 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:16.132984 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:16.133580 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:41:16.133600 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:41:16.135132 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:41:16.136812 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:41:16.136818 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:41:16.136834 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:41:16.137093 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:41:16.137348 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:41:16.137351 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:41:16.137353 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:41:16.137493 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:41:16.137858 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:41:16.137864 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:41:16.138123 (+ 259us) client_negotiation.cc:769] Sending connection context
0619 14:41:16.138299 (+ 176us) client_negotiation.cc:240] Negotiation successful
0619 14:41:16.138625 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":103,"threads_started":1}
I20260619 14:41:16.140611 17638 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42301
I20260619 14:41:16.140884 17638 heartbeater.cc:464] Registering TS with master...
I20260619 14:41:16.141438 17638 heartbeater.cc:511] Master 127.0.98.62:42301 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 14:41:16.143050 17447 ts_manager.cc:194] Registered new tserver with Master: 97a837f775c5419690c4e49710fce43f (127.0.98.1:40773)
I20260619 14:41:16.144189 17447 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49111
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 14:41:16.157562 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42301
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:41:16.264820 17643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:41:16.265090 17643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:41:16.265132 17643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:41:16.269156 17643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:41:16.269219 17643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:41:16.269281 17643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:41:16.274544 17643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:41:16.275576 17643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:41:16.276557 17643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:41:16.285935 17649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:16.285914 17648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:16.286017 17643 server_base.cc:1034] running on GCE node
W20260619 14:41:16.285908 17651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:16.286609 17643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:41:16.287247 17643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:41:16.288393 17643 hybrid_clock.cc:648] HybridClock initialized: now 1781880076288368 us; error 38 us; skew 500 ppm
Jun 19 14:41:16 dist-test-slave-32hj krb5kdc[17402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:41:16.291059 17643 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:41:16.291947 17643 webserver.cc:460] Webserver started at http://127.0.98.2:46069/ using document root <none> and password file <none>
I20260619 14:41:16.292482 17643 fs_manager.cc:359] Metadata directory not provided
I20260619 14:41:16.292547 17643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:41:16.292771 17643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:41:16.294809 17643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/ts-1/data/instance:
uuid: "cd42798bc8314a3d84080e39bedcb7e7"
format_stamp: "Formatted at 2026-06-19 14:41:16 on dist-test-slave-32hj"
server_key: "80d239bc465967295356985e58997be1"
server_key_iv: "d76e6cfc08f8beee4f728e93389589a3"
server_key_version: "encryptionkey@0"
I20260619 14:41:16.295389 17643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/ts-1/wal/instance:
uuid: "cd42798bc8314a3d84080e39bedcb7e7"
format_stamp: "Formatted at 2026-06-19 14:41:16 on dist-test-slave-32hj"
server_key: "80d239bc465967295356985e58997be1"
server_key_iv: "d76e6cfc08f8beee4f728e93389589a3"
server_key_version: "encryptionkey@0"
I20260619 14:41:16.299151 17643 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:41:16.301551 17658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:16.302815 17643 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:41:16.302937 17643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/ts-1/wal
uuid: "cd42798bc8314a3d84080e39bedcb7e7"
format_stamp: "Formatted at 2026-06-19 14:41:16 on dist-test-slave-32hj"
server_key: "80d239bc465967295356985e58997be1"
server_key_iv: "d76e6cfc08f8beee4f728e93389589a3"
server_key_version: "encryptionkey@0"
I20260619 14:41:16.303035 17643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:41:16.320063 17643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:41:16.323158 17643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:41:16.323331 17643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:41:16.323894 17643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:41:16.324684 17643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:41:16.324762 17643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:16.324810 17643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:41:16.324831 17643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:16.333176 17643 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40121
I20260619 14:41:16.333204 17771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40121 every 8 connection(s)
I20260619 14:41:16.334024 17643 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 14:41:16 dist-test-slave-32hj krb5kdc[17402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:41:16.343870 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 17643
I20260619 14:41:16.343988 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:41:16.344255 392 external_mini_cluster.cc:1428] Setting key aaf813966c734d03797cb27472b351cb
I20260619 14:41:16.346608 17486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:16.335676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58545 (local address 127.0.98.62:42301)
0619 14:41:16.335795 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:16.335798 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:16.336837 (+ 1039us) server_negotiation.cc:407] Connection header received
0619 14:41:16.337700 (+ 863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:16.337703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:16.337770 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:16.337841 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:16.338969 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:16.339494 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:16.340588 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:16.340868 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:16.340954 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:16.343212 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:16.343216 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:16.343219 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:41:16.343245 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:16.344566 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:16.345115 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:16.345119 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:16.345121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:16.345204 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:16.345567 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:16.345570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:16.345571 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:16.345771 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:16.345859 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:16.346340 (+ 481us) server_negotiation.cc:299] Negotiation successful
0619 14:41:16.346456 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 14:41:16.347627 17774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:16.336015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42301 (local address 127.0.98.2:58545)
0619 14:41:16.336675 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:16.336734 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:16.337508 (+ 774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:16.338008 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:16.338015 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:16.338376 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:41:16.338834 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:16.338847 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:16.339605 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:16.339607 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:16.340478 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:16.340486 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:16.341010 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:16.341014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:16.341166 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:16.341686 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:41:16.341702 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:41:16.343054 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:41:16.344672 (+ 1618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:41:16.344677 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:41:16.344689 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:41:16.345027 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:41:16.345291 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:41:16.345293 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:41:16.345294 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:41:16.345480 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:41:16.345869 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:41:16.345876 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:41:16.346115 (+ 239us) client_negotiation.cc:769] Sending connection context
0619 14:41:16.346288 (+ 173us) client_negotiation.cc:240] Negotiation successful
0619 14:41:16.346608 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":105,"threads_started":1}
I20260619 14:41:16.348785 17772 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42301
I20260619 14:41:16.349036 17772 heartbeater.cc:464] Registering TS with master...
I20260619 14:41:16.349516 17772 heartbeater.cc:511] Master 127.0.98.62:42301 requested a full tablet report, sending...
I20260619 14:41:16.350522 17447 ts_manager.cc:194] Registered new tserver with Master: cd42798bc8314a3d84080e39bedcb7e7 (127.0.98.2:40121)
I20260619 14:41:16.351050 17447 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58545
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 14:41:16.371037 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42301
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:41:16.480654 17777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:41:16.480968 17777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:41:16.481014 17777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:41:16.485001 17777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:41:16.485059 17777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:41:16.485116 17777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:41:16.490442 17777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:41:16.491479 17777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:41:16.492452 17777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:41:16.501969 17782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:16.501977 17785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:16.502125 17777 server_base.cc:1034] running on GCE node
W20260619 14:41:16.501960 17783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:16.502657 17777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:41:16.503299 17777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:41:16.504482 17777 hybrid_clock.cc:648] HybridClock initialized: now 1781880076504460 us; error 33 us; skew 500 ppm
Jun 19 14:41:16 dist-test-slave-32hj krb5kdc[17402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:41:16.507439 17777 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:41:16.508349 17777 webserver.cc:460] Webserver started at http://127.0.98.3:33695/ using document root <none> and password file <none>
I20260619 14:41:16.508956 17777 fs_manager.cc:359] Metadata directory not provided
I20260619 14:41:16.509029 17777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:41:16.509246 17777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:41:16.511340 17777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/ts-2/data/instance:
uuid: "86173467c5574c6c89bd9de1b791f3e1"
format_stamp: "Formatted at 2026-06-19 14:41:16 on dist-test-slave-32hj"
server_key: "5713c6e931f54231c9a0f0b03d4c06a6"
server_key_iv: "dc9bd9b25d6220c2c1830a34efd74d91"
server_key_version: "encryptionkey@0"
I20260619 14:41:16.511889 17777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/ts-2/wal/instance:
uuid: "86173467c5574c6c89bd9de1b791f3e1"
format_stamp: "Formatted at 2026-06-19 14:41:16 on dist-test-slave-32hj"
server_key: "5713c6e931f54231c9a0f0b03d4c06a6"
server_key_iv: "dc9bd9b25d6220c2c1830a34efd74d91"
server_key_version: "encryptionkey@0"
I20260619 14:41:16.515748 17777 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260619 14:41:16.518301 17792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:16.519519 17777 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260619 14:41:16.519632 17777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/ts-2/wal
uuid: "86173467c5574c6c89bd9de1b791f3e1"
format_stamp: "Formatted at 2026-06-19 14:41:16 on dist-test-slave-32hj"
server_key: "5713c6e931f54231c9a0f0b03d4c06a6"
server_key_iv: "dc9bd9b25d6220c2c1830a34efd74d91"
server_key_version: "encryptionkey@0"
I20260619 14:41:16.519734 17777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:41:16.544435 17777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:41:16.548009 17777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:41:16.548223 17777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:41:16.548889 17777 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:41:16.549763 17777 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:41:16.549813 17777 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:16.549865 17777 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:41:16.549894 17777 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:16.558756 17777 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39059
I20260619 14:41:16.558777 17905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39059 every 8 connection(s)
I20260619 14:41:16.559688 17777 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:41:16.567896 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 17777
I20260619 14:41:16.568046 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:41:16.568354 392 external_mini_cluster.cc:1428] Setting key 7d39ecc31bdf681be38ada9a17662c8c
Jun 19 14:41:16 dist-test-slave-32hj krb5kdc[17402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:41:16.574023 17486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:16.561552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40685 (local address 127.0.98.62:42301)
0619 14:41:16.561724 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:16.561729 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:16.562628 (+ 899us) server_negotiation.cc:407] Connection header received
0619 14:41:16.563957 (+ 1329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:16.563960 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:16.564012 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:16.564094 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:16.565473 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:16.566027 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:16.567278 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:16.567511 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:16.567610 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:16.570649 (+ 3039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:16.570653 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:16.570656 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:41:16.570687 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:16.571994 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:16.572500 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:16.572503 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:16.572504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:16.572570 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:16.572893 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:16.572895 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:16.572896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:16.573074 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:16.573150 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:16.573781 (+ 631us) server_negotiation.cc:299] Negotiation successful
0619 14:41:16.573898 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 14:41:16.574903 17908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:16.561974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42301 (local address 127.0.98.3:40685)
0619 14:41:16.562458 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:16.562504 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:16.563728 (+ 1224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:16.564288 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:16.564297 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:16.564699 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:41:16.565282 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:16.565297 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:16.566183 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:16.566186 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:16.567141 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:16.567151 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:16.567729 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:16.567734 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:16.567981 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:16.568695 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:41:16.568712 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:41:16.570446 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:41:16.572117 (+ 1671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:41:16.572122 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:41:16.572137 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:41:16.572417 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:41:16.572631 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:41:16.572633 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:41:16.572634 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:41:16.572828 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:41:16.573189 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:41:16.573196 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:41:16.573496 (+ 300us) client_negotiation.cc:769] Sending connection context
0619 14:41:16.573684 (+ 188us) client_negotiation.cc:240] Negotiation successful
0619 14:41:16.573985 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":111,"threads_started":1}
I20260619 14:41:16.576241 17906 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42301
I20260619 14:41:16.576489 17906 heartbeater.cc:464] Registering TS with master...
I20260619 14:41:16.577085 17906 heartbeater.cc:511] Master 127.0.98.62:42301 requested a full tablet report, sending...
I20260619 14:41:16.578229 17447 ts_manager.cc:194] Registered new tserver with Master: 86173467c5574c6c89bd9de1b791f3e1 (127.0.98.3:39059)
I20260619 14:41:16.578857 17447 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40685
I20260619 14:41:16.582728 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:41:16.591832 17486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:16.584632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60082 (local address 127.0.98.62:42301)
0619 14:41:16.584813 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:16.584818 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:16.584969 (+ 151us) server_negotiation.cc:407] Connection header received
0619 14:41:16.585091 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:16.585093 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:16.585135 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:16.585200 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:16.585853 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:16.586364 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:16.587188 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:16.587388 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:16.587475 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:16.589010 (+ 1535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:16.589013 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:16.589015 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:41:16.589056 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:16.590359 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:16.590753 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:16.590755 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:16.590756 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:16.590825 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:16.591044 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:16.591046 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:16.591047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:16.591253 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:16.591354 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:16.591613 (+ 259us) server_negotiation.cc:299] Negotiation successful
0619 14:41:16.591710 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 14:41:16.594741 17447 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60082:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:41:16.597836 17447 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:41:16.612709 17918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:16.606196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54256 (local address 127.0.98.1:40773)
0619 14:41:16.607527 (+ 1331us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:16.607542 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:16.607564 (+ 22us) server_negotiation.cc:407] Connection header received
0619 14:41:16.607631 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:16.607635 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:16.607843 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:16.608016 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:41:16.608212 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:16.609559 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:16.611142 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:16.612011 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:16.612120 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:16.612264 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:16.612444 (+ 180us) server_negotiation.cc:299] Negotiation successful
0619 14:41:16.612545 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1229,"thread_start_us":83,"threads_started":1}
I20260619 14:41:16.612704 17917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:16.606009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40773 (local address 127.0.0.1:54256)
0619 14:41:16.606677 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:16.606732 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:16.606860 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:16.607986 (+ 1126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:16.607990 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:16.608007 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:41:16.608122 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:16.608130 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:16.609690 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:16.609693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:16.611020 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:16.611030 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:16.612137 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:16.612141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:16.612252 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:16.612289 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 14:41:16.612403 (+ 114us) client_negotiation.cc:240] Negotiation successful
0619 14:41:16.612540 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"mutex_wait_us":69,"thread_start_us":92,"threads_started":1}
I20260619 14:41:16.613360 17920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:16.606217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41644 (local address 127.0.98.2:40121)
0619 14:41:16.606791 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:16.606803 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:16.607116 (+ 313us) server_negotiation.cc:407] Connection header received
0619 14:41:16.607200 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:16.607205 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:16.607448 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:16.607601 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:41:16.608083 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:16.609398 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:16.611199 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:16.612011 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:16.612120 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:16.612264 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:16.613102 (+ 838us) server_negotiation.cc:299] Negotiation successful
0619 14:41:16.613201 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":100,"threads_started":1}
I20260619 14:41:16.613628 17922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:16.607114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45926 (local address 127.0.98.3:39059)
0619 14:41:16.607527 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:16.607541 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:16.607563 (+ 22us) server_negotiation.cc:407] Connection header received
0619 14:41:16.607633 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:16.607636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:16.607773 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:16.608155 (+ 382us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:41:16.608503 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:16.609301 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:16.611566 (+ 2265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:16.612139 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:16.612238 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:16.612376 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:16.612797 (+ 421us) server_negotiation.cc:299] Negotiation successful
0619 14:41:16.613323 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":87,"threads_started":1}
I20260619 14:41:16.613698 17919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:16.606009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40121 (local address 127.0.0.1:41644)
0619 14:41:16.606931 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:16.606942 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:16.607028 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:16.607788 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:16.607794 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:16.607815 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:41:16.607963 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:16.607971 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:16.609527 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:16.609530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:16.611080 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:16.611091 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:16.612360 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:16.612363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:16.612467 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:16.612491 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 14:41:16.612555 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 14:41:16.612627 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":861,"mutex_wait_us":278,"thread_start_us":116,"threads_started":1}
I20260619 14:41:16.613961 17921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:16.606701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39059 (local address 127.0.0.1:45926)
0619 14:41:16.607243 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:16.607255 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:16.607330 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:16.608298 (+ 968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:16.608300 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:16.608314 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:41:16.608382 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:16.608386 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:16.609428 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:16.609431 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:16.611460 (+ 2029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:16.611468 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:16.612436 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:16.612439 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:16.612546 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:16.612566 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 14:41:16.612629 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 14:41:16.612690 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"mutex_wait_us":118,"thread_start_us":103,"threads_started":1}
I20260619 14:41:16.616047 17572 tablet_service.cc:1467] Processing CreateTablet for tablet 907080abf8ed443db1fd04f351d7ed6f (DEFAULT_TABLE table=test-table [id=c85e6f9fd9df4e52af9c79089e7ba997]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:41:16.616456 17840 tablet_service.cc:1467] Processing CreateTablet for tablet 907080abf8ed443db1fd04f351d7ed6f (DEFAULT_TABLE table=test-table [id=c85e6f9fd9df4e52af9c79089e7ba997]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:41:16.616456 17706 tablet_service.cc:1467] Processing CreateTablet for tablet 907080abf8ed443db1fd04f351d7ed6f (DEFAULT_TABLE table=test-table [id=c85e6f9fd9df4e52af9c79089e7ba997]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:41:16.617220 17572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 907080abf8ed443db1fd04f351d7ed6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:41:16.617498 17706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 907080abf8ed443db1fd04f351d7ed6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:41:16.617498 17840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 907080abf8ed443db1fd04f351d7ed6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:41:16.624179 17923 tablet_bootstrap.cc:492] T 907080abf8ed443db1fd04f351d7ed6f P 97a837f775c5419690c4e49710fce43f: Bootstrap starting.
I20260619 14:41:16.624996 17925 tablet_bootstrap.cc:492] T 907080abf8ed443db1fd04f351d7ed6f P 86173467c5574c6c89bd9de1b791f3e1: Bootstrap starting.
I20260619 14:41:16.626763 17925 tablet_bootstrap.cc:654] T 907080abf8ed443db1fd04f351d7ed6f P 86173467c5574c6c89bd9de1b791f3e1: Neither blocks nor log segments found. Creating new log.
I20260619 14:41:16.626765 17923 tablet_bootstrap.cc:654] T 907080abf8ed443db1fd04f351d7ed6f P 97a837f775c5419690c4e49710fce43f: Neither blocks nor log segments found. Creating new log.
I20260619 14:41:16.627738 17925 log.cc:824] T 907080abf8ed443db1fd04f351d7ed6f P 86173467c5574c6c89bd9de1b791f3e1: Log is configured to *not* fsync() on all Append() calls
I20260619 14:41:16.627794 17923 log.cc:824] T 907080abf8ed443db1fd04f351d7ed6f P 97a837f775c5419690c4e49710fce43f: Log is configured to *not* fsync() on all Append() calls
I20260619 14:41:16.628350 17924 tablet_bootstrap.cc:492] T 907080abf8ed443db1fd04f351d7ed6f P cd42798bc8314a3d84080e39bedcb7e7: Bootstrap starting.
I20260619 14:41:16.629737 17925 tablet_bootstrap.cc:492] T 907080abf8ed443db1fd04f351d7ed6f P 86173467c5574c6c89bd9de1b791f3e1: No bootstrap required, opened a new log
I20260619 14:41:16.629793 17923 tablet_bootstrap.cc:492] T 907080abf8ed443db1fd04f351d7ed6f P 97a837f775c5419690c4e49710fce43f: No bootstrap required, opened a new log
I20260619 14:41:16.629931 17925 ts_tablet_manager.cc:1397] T 907080abf8ed443db1fd04f351d7ed6f P 86173467c5574c6c89bd9de1b791f3e1: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260619 14:41:16.629962 17923 ts_tablet_manager.cc:1397] T 907080abf8ed443db1fd04f351d7ed6f P 97a837f775c5419690c4e49710fce43f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 14:41:16.630798 17924 tablet_bootstrap.cc:654] T 907080abf8ed443db1fd04f351d7ed6f P cd42798bc8314a3d84080e39bedcb7e7: Neither blocks nor log segments found. Creating new log.
I20260619 14:41:16.631569 17924 log.cc:824] T 907080abf8ed443db1fd04f351d7ed6f P cd42798bc8314a3d84080e39bedcb7e7: Log is configured to *not* fsync() on all Append() calls
I20260619 14:41:16.633285 17925 raft_consensus.cc:348] T 907080abf8ed443db1fd04f351d7ed6f P 86173467c5574c6c89bd9de1b791f3e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97a837f775c5419690c4e49710fce43f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40773 } } peers { permanent_uuid: "cd42798bc8314a3d84080e39bedcb7e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40121 } } peers { permanent_uuid: "86173467c5574c6c89bd9de1b791f3e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } }
I20260619 14:41:16.633517 17925 raft_consensus.cc:374] T 907080abf8ed443db1fd04f351d7ed6f P 86173467c5574c6c89bd9de1b791f3e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:41:16.633574 17925 raft_consensus.cc:729] T 907080abf8ed443db1fd04f351d7ed6f P 86173467c5574c6c89bd9de1b791f3e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86173467c5574c6c89bd9de1b791f3e1, State: Initialized, Role: FOLLOWER
I20260619 14:41:16.633644 17924 tablet_bootstrap.cc:492] T 907080abf8ed443db1fd04f351d7ed6f P cd42798bc8314a3d84080e39bedcb7e7: No bootstrap required, opened a new log
I20260619 14:41:16.633487 17923 raft_consensus.cc:348] T 907080abf8ed443db1fd04f351d7ed6f P 97a837f775c5419690c4e49710fce43f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97a837f775c5419690c4e49710fce43f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40773 } } peers { permanent_uuid: "cd42798bc8314a3d84080e39bedcb7e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40121 } } peers { permanent_uuid: "86173467c5574c6c89bd9de1b791f3e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } }
I20260619 14:41:16.633713 17923 raft_consensus.cc:374] T 907080abf8ed443db1fd04f351d7ed6f P 97a837f775c5419690c4e49710fce43f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:41:16.633782 17923 raft_consensus.cc:729] T 907080abf8ed443db1fd04f351d7ed6f P 97a837f775c5419690c4e49710fce43f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97a837f775c5419690c4e49710fce43f, State: Initialized, Role: FOLLOWER
I20260619 14:41:16.633822 17924 ts_tablet_manager.cc:1397] T 907080abf8ed443db1fd04f351d7ed6f P cd42798bc8314a3d84080e39bedcb7e7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 14:41:16.634073 17925 consensus_queue.cc:260] T 907080abf8ed443db1fd04f351d7ed6f P 86173467c5574c6c89bd9de1b791f3e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97a837f775c5419690c4e49710fce43f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40773 } } peers { permanent_uuid: "cd42798bc8314a3d84080e39bedcb7e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40121 } } peers { permanent_uuid: "86173467c5574c6c89bd9de1b791f3e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } }
I20260619 14:41:16.634181 17923 consensus_queue.cc:260] T 907080abf8ed443db1fd04f351d7ed6f P 97a837f775c5419690c4e49710fce43f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97a837f775c5419690c4e49710fce43f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40773 } } peers { permanent_uuid: "cd42798bc8314a3d84080e39bedcb7e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40121 } } peers { permanent_uuid: "86173467c5574c6c89bd9de1b791f3e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } }
I20260619 14:41:16.634696 17906 heartbeater.cc:503] Master 127.0.98.62:42301 was elected leader, sending a full tablet report...
I20260619 14:41:16.635063 17923 ts_tablet_manager.cc:1428] T 907080abf8ed443db1fd04f351d7ed6f P 97a837f775c5419690c4e49710fce43f: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 14:41:16.635077 17925 ts_tablet_manager.cc:1428] T 907080abf8ed443db1fd04f351d7ed6f P 86173467c5574c6c89bd9de1b791f3e1: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 14:41:16.635390 17638 heartbeater.cc:503] Master 127.0.98.62:42301 was elected leader, sending a full tablet report...
I20260619 14:41:16.638311 17924 raft_consensus.cc:348] T 907080abf8ed443db1fd04f351d7ed6f P cd42798bc8314a3d84080e39bedcb7e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97a837f775c5419690c4e49710fce43f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40773 } } peers { permanent_uuid: "cd42798bc8314a3d84080e39bedcb7e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40121 } } peers { permanent_uuid: "86173467c5574c6c89bd9de1b791f3e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } }
I20260619 14:41:16.638561 17924 raft_consensus.cc:374] T 907080abf8ed443db1fd04f351d7ed6f P cd42798bc8314a3d84080e39bedcb7e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:41:16.638618 17924 raft_consensus.cc:729] T 907080abf8ed443db1fd04f351d7ed6f P cd42798bc8314a3d84080e39bedcb7e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd42798bc8314a3d84080e39bedcb7e7, State: Initialized, Role: FOLLOWER
I20260619 14:41:16.639021 17924 consensus_queue.cc:260] T 907080abf8ed443db1fd04f351d7ed6f P cd42798bc8314a3d84080e39bedcb7e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97a837f775c5419690c4e49710fce43f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40773 } } peers { permanent_uuid: "cd42798bc8314a3d84080e39bedcb7e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40121 } } peers { permanent_uuid: "86173467c5574c6c89bd9de1b791f3e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } }
I20260619 14:41:16.639621 17772 heartbeater.cc:503] Master 127.0.98.62:42301 was elected leader, sending a full tablet report...
I20260619 14:41:16.639946 17924 ts_tablet_manager.cc:1428] T 907080abf8ed443db1fd04f351d7ed6f P cd42798bc8314a3d84080e39bedcb7e7: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 14:41:16.738600 17929 raft_consensus.cc:482] T 907080abf8ed443db1fd04f351d7ed6f P 86173467c5574c6c89bd9de1b791f3e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:41:16.738812 17929 raft_consensus.cc:504] T 907080abf8ed443db1fd04f351d7ed6f P 86173467c5574c6c89bd9de1b791f3e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97a837f775c5419690c4e49710fce43f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40773 } } peers { permanent_uuid: "cd42798bc8314a3d84080e39bedcb7e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40121 } } peers { permanent_uuid: "86173467c5574c6c89bd9de1b791f3e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } }
I20260619 14:41:16.739955 17929 leader_election.cc:288] T 907080abf8ed443db1fd04f351d7ed6f P 86173467c5574c6c89bd9de1b791f3e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97a837f775c5419690c4e49710fce43f (127.0.98.1:40773), cd42798bc8314a3d84080e39bedcb7e7 (127.0.98.2:40121)
I20260619 14:41:16.743664 17918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:16.740269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41867 (local address 127.0.98.1:40773)
0619 14:41:16.740375 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:16.740380 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:16.740448 (+ 68us) server_negotiation.cc:407] Connection header received
0619 14:41:16.740620 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:16.740624 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:16.740677 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:16.740788 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:41:16.741053 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:16.741619 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:16.742765 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:16.743138 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:16.743214 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:16.743279 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:16.743465 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 14:41:16.743531 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 14:41:16.743671 17920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:16.740355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41783 (local address 127.0.98.2:40121)
0619 14:41:16.740492 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:16.740496 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:16.740695 (+ 199us) server_negotiation.cc:407] Connection header received
0619 14:41:16.740830 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:16.740832 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:16.740876 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:16.740962 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:41:16.741199 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:16.741673 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:16.742755 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:16.743171 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:16.743266 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:16.743326 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:16.743489 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 14:41:16.743549 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 14:41:16.743671 17908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:16.740179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40773 (local address 127.0.98.3:41867)
0619 14:41:16.740360 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:16.740374 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:16.740490 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:16.740800 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:16.740803 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:16.740822 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:41:16.740952 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:16.740960 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:16.741726 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:16.741730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:16.742642 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:16.742651 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:16.743249 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:16.743253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:16.743350 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:16.743373 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 14:41:16.743440 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 14:41:16.743508 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":68}
I20260619 14:41:16.743686 17932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:16.740252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40121 (local address 127.0.98.3:41783)
0619 14:41:16.740602 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:16.740614 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:16.740712 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:16.740977 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:16.740979 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:16.740993 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:41:16.741104 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:16.741110 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:16.741775 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:16.741777 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:16.742637 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:16.742645 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:16.743286 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:16.743290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:16.743378 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:16.743395 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 14:41:16.743464 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 14:41:16.743531 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":3,"thread_start_us":133,"threads_started":1}
I20260619 14:41:16.744306 17592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907080abf8ed443db1fd04f351d7ed6f" candidate_uuid: "86173467c5574c6c89bd9de1b791f3e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97a837f775c5419690c4e49710fce43f" is_pre_election: true
I20260619 14:41:16.744422 17726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907080abf8ed443db1fd04f351d7ed6f" candidate_uuid: "86173467c5574c6c89bd9de1b791f3e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd42798bc8314a3d84080e39bedcb7e7" is_pre_election: true
I20260619 14:41:16.744611 17592 raft_consensus.cc:2454] T 907080abf8ed443db1fd04f351d7ed6f P 97a837f775c5419690c4e49710fce43f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86173467c5574c6c89bd9de1b791f3e1 in term 0.
I20260619 14:41:16.744690 17726 raft_consensus.cc:2454] T 907080abf8ed443db1fd04f351d7ed6f P cd42798bc8314a3d84080e39bedcb7e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86173467c5574c6c89bd9de1b791f3e1 in term 0.
I20260619 14:41:16.745105 17796 leader_election.cc:302] T 907080abf8ed443db1fd04f351d7ed6f P 86173467c5574c6c89bd9de1b791f3e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86173467c5574c6c89bd9de1b791f3e1, 97a837f775c5419690c4e49710fce43f; no voters:
I20260619 14:41:16.745361 17929 raft_consensus.cc:2781] T 907080abf8ed443db1fd04f351d7ed6f P 86173467c5574c6c89bd9de1b791f3e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:41:16.745447 17929 raft_consensus.cc:482] T 907080abf8ed443db1fd04f351d7ed6f P 86173467c5574c6c89bd9de1b791f3e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:41:16.745496 17929 raft_consensus.cc:3037] T 907080abf8ed443db1fd04f351d7ed6f P 86173467c5574c6c89bd9de1b791f3e1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:41:16.746465 17929 raft_consensus.cc:504] T 907080abf8ed443db1fd04f351d7ed6f P 86173467c5574c6c89bd9de1b791f3e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97a837f775c5419690c4e49710fce43f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40773 } } peers { permanent_uuid: "cd42798bc8314a3d84080e39bedcb7e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40121 } } peers { permanent_uuid: "86173467c5574c6c89bd9de1b791f3e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } }
I20260619 14:41:16.746773 17929 leader_election.cc:288] T 907080abf8ed443db1fd04f351d7ed6f P 86173467c5574c6c89bd9de1b791f3e1 [CANDIDATE]: Term 1 election: Requested vote from peers 97a837f775c5419690c4e49710fce43f (127.0.98.1:40773), cd42798bc8314a3d84080e39bedcb7e7 (127.0.98.2:40121)
I20260619 14:41:16.747097 17592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907080abf8ed443db1fd04f351d7ed6f" candidate_uuid: "86173467c5574c6c89bd9de1b791f3e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97a837f775c5419690c4e49710fce43f"
I20260619 14:41:16.747124 17726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907080abf8ed443db1fd04f351d7ed6f" candidate_uuid: "86173467c5574c6c89bd9de1b791f3e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd42798bc8314a3d84080e39bedcb7e7"
I20260619 14:41:16.747221 17592 raft_consensus.cc:3037] T 907080abf8ed443db1fd04f351d7ed6f P 97a837f775c5419690c4e49710fce43f [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:41:16.747221 17726 raft_consensus.cc:3037] T 907080abf8ed443db1fd04f351d7ed6f P cd42798bc8314a3d84080e39bedcb7e7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:41:16.748191 17592 raft_consensus.cc:2454] T 907080abf8ed443db1fd04f351d7ed6f P 97a837f775c5419690c4e49710fce43f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86173467c5574c6c89bd9de1b791f3e1 in term 1.
I20260619 14:41:16.748217 17726 raft_consensus.cc:2454] T 907080abf8ed443db1fd04f351d7ed6f P cd42798bc8314a3d84080e39bedcb7e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86173467c5574c6c89bd9de1b791f3e1 in term 1.
I20260619 14:41:16.748472 17794 leader_election.cc:302] T 907080abf8ed443db1fd04f351d7ed6f P 86173467c5574c6c89bd9de1b791f3e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86173467c5574c6c89bd9de1b791f3e1, cd42798bc8314a3d84080e39bedcb7e7; no voters:
I20260619 14:41:16.748627 17929 raft_consensus.cc:2781] T 907080abf8ed443db1fd04f351d7ed6f P 86173467c5574c6c89bd9de1b791f3e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:41:16.748871 17929 raft_consensus.cc:686] T 907080abf8ed443db1fd04f351d7ed6f P 86173467c5574c6c89bd9de1b791f3e1 [term 1 LEADER]: Becoming Leader. State: Replica: 86173467c5574c6c89bd9de1b791f3e1, State: Running, Role: LEADER
I20260619 14:41:16.749164 17929 consensus_queue.cc:237] T 907080abf8ed443db1fd04f351d7ed6f P 86173467c5574c6c89bd9de1b791f3e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97a837f775c5419690c4e49710fce43f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40773 } } peers { permanent_uuid: "cd42798bc8314a3d84080e39bedcb7e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40121 } } peers { permanent_uuid: "86173467c5574c6c89bd9de1b791f3e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } }
I20260619 14:41:16.752094 17447 catalog_manager.cc:5495] T 907080abf8ed443db1fd04f351d7ed6f P 86173467c5574c6c89bd9de1b791f3e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 86173467c5574c6c89bd9de1b791f3e1 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "86173467c5574c6c89bd9de1b791f3e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97a837f775c5419690c4e49710fce43f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd42798bc8314a3d84080e39bedcb7e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "86173467c5574c6c89bd9de1b791f3e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } health_report { overall_health: HEALTHY } } }
I20260619 14:41:16.763450 17922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:16.760037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45930 (local address 127.0.98.3:39059)
0619 14:41:16.760163 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:16.760168 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:16.760182 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:41:16.760323 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:16.760325 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:16.760382 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:16.760470 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:41:16.760706 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:16.761232 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:16.762055 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:16.762266 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:16.762351 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:16.762637 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:41:16.763059 (+ 422us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:41:16.763165 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:16.763232 (+ 67us) server_negotiation.cc:299] Negotiation successful
0619 14:41:16.763290 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 14:41:16.769229 17592 raft_consensus.cc:1261] T 907080abf8ed443db1fd04f351d7ed6f P 97a837f775c5419690c4e49710fce43f [term 1 FOLLOWER]: Refusing update from remote peer 86173467c5574c6c89bd9de1b791f3e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:41:16.769229 17726 raft_consensus.cc:1261] T 907080abf8ed443db1fd04f351d7ed6f P cd42798bc8314a3d84080e39bedcb7e7 [term 1 FOLLOWER]: Refusing update from remote peer 86173467c5574c6c89bd9de1b791f3e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:41:16.769954 17933 consensus_queue.cc:1035] T 907080abf8ed443db1fd04f351d7ed6f P 86173467c5574c6c89bd9de1b791f3e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97a837f775c5419690c4e49710fce43f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:41:16.770126 17929 consensus_queue.cc:1035] T 907080abf8ed443db1fd04f351d7ed6f P 86173467c5574c6c89bd9de1b791f3e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd42798bc8314a3d84080e39bedcb7e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:41:16.776629 17936 mvcc.cc:205] Tried to move back new op lower bound from 7298580794439315456 to 7298580794367115264. Current Snapshot: MvccSnapshot[applied={T|T < 7298580794439315456}]
I20260619 14:41:16.777346 17938 mvcc.cc:205] Tried to move back new op lower bound from 7298580794439315456 to 7298580794367115264. Current Snapshot: MvccSnapshot[applied={T|T < 7298580794439315456}]
I20260619 14:41:16.778224 17937 mvcc.cc:205] Tried to move back new op lower bound from 7298580794439315456 to 7298580794367115264. Current Snapshot: MvccSnapshot[applied={T|T < 7298580794439315456}]
I20260619 14:41:16.787423 17447 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:41:16.787587 17447 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:41:16.790162 17447 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 79df4cbd968a412fac2675ea730ad180: Sending DeleteTablet for 3 replicas of tablet 907080abf8ed443db1fd04f351d7ed6f
I20260619 14:41:16.791079 17840 tablet_service.cc:1514] Processing DeleteTablet for tablet 907080abf8ed443db1fd04f351d7ed6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:41:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45926
I20260619 14:41:16.791049 17572 tablet_service.cc:1514] Processing DeleteTablet for tablet 907080abf8ed443db1fd04f351d7ed6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:41:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54256
I20260619 14:41:16.791116 17706 tablet_service.cc:1514] Processing DeleteTablet for tablet 907080abf8ed443db1fd04f351d7ed6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:41:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41644
I20260619 14:41:16.791594 17947 tablet_replica.cc:330] T 907080abf8ed443db1fd04f351d7ed6f P 97a837f775c5419690c4e49710fce43f: stopping tablet replica
I20260619 14:41:16.791900 17946 tablet_replica.cc:330] T 907080abf8ed443db1fd04f351d7ed6f P cd42798bc8314a3d84080e39bedcb7e7: stopping tablet replica
I20260619 14:41:16.791949 17947 raft_consensus.cc:2229] T 907080abf8ed443db1fd04f351d7ed6f P 97a837f775c5419690c4e49710fce43f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:41:16.792212 17946 raft_consensus.cc:2229] T 907080abf8ed443db1fd04f351d7ed6f P cd42798bc8314a3d84080e39bedcb7e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:41:16.792306 17947 raft_consensus.cc:2258] T 907080abf8ed443db1fd04f351d7ed6f P 97a837f775c5419690c4e49710fce43f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:41:16.792518 17946 raft_consensus.cc:2258] T 907080abf8ed443db1fd04f351d7ed6f P cd42798bc8314a3d84080e39bedcb7e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:41:16.792644 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 17509
I20260619 14:41:16.794165 17946 ts_tablet_manager.cc:1905] T 907080abf8ed443db1fd04f351d7ed6f P cd42798bc8314a3d84080e39bedcb7e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:41:16.795083 17945 tablet_replica.cc:330] T 907080abf8ed443db1fd04f351d7ed6f P 86173467c5574c6c89bd9de1b791f3e1: stopping tablet replica
I20260619 14:41:16.795425 17945 raft_consensus.cc:2229] T 907080abf8ed443db1fd04f351d7ed6f P 86173467c5574c6c89bd9de1b791f3e1 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:41:16.795883 17945 raft_consensus.cc:2258] T 907080abf8ed443db1fd04f351d7ed6f P 86173467c5574c6c89bd9de1b791f3e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:41:16.797179 17946 ts_tablet_manager.cc:1918] T 907080abf8ed443db1fd04f351d7ed6f P cd42798bc8314a3d84080e39bedcb7e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:41:16.797278 17946 log.cc:1196] T 907080abf8ed443db1fd04f351d7ed6f P cd42798bc8314a3d84080e39bedcb7e7: Deleting WAL directory at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/ts-1/wal/wals/907080abf8ed443db1fd04f351d7ed6f
I20260619 14:41:16.797556 17945 ts_tablet_manager.cc:1905] T 907080abf8ed443db1fd04f351d7ed6f P 86173467c5574c6c89bd9de1b791f3e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:41:16.797586 17946 ts_tablet_manager.cc:1939] T 907080abf8ed443db1fd04f351d7ed6f P cd42798bc8314a3d84080e39bedcb7e7: Deleting consensus metadata
I20260619 14:41:16.799042 17432 catalog_manager.cc:4841] TS cd42798bc8314a3d84080e39bedcb7e7 (127.0.98.2:40121): tablet 907080abf8ed443db1fd04f351d7ed6f (table test-table [id=c85e6f9fd9df4e52af9c79089e7ba997]) successfully deleted
I20260619 14:41:16.800200 17945 ts_tablet_manager.cc:1918] T 907080abf8ed443db1fd04f351d7ed6f P 86173467c5574c6c89bd9de1b791f3e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:41:16.800356 17945 log.cc:1196] T 907080abf8ed443db1fd04f351d7ed6f P 86173467c5574c6c89bd9de1b791f3e1: Deleting WAL directory at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781879837648036-392-0/minicluster-data/ts-2/wal/wals/907080abf8ed443db1fd04f351d7ed6f
W20260619 14:41:16.800366 17434 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40773 (error 108)
I20260619 14:41:16.800457 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 17643
I20260619 14:41:16.800817 17945 ts_tablet_manager.cc:1939] T 907080abf8ed443db1fd04f351d7ed6f P 86173467c5574c6c89bd9de1b791f3e1: Deleting consensus metadata
I20260619 14:41:16.801781 17921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:16.801460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40773 (local address 127.0.0.1:54268)
0619 14:41:16.801630 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:16.801696 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260619 14:41:16.801937 17434 catalog_manager.cc:4584] TS 97a837f775c5419690c4e49710fce43f (127.0.98.1:40773): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 907080abf8ed443db1fd04f351d7ed6f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40773: connect: Connection refused (error 111)
I20260619 14:41:16.801973 17431 catalog_manager.cc:4841] TS 86173467c5574c6c89bd9de1b791f3e1 (127.0.98.3:39059): tablet 907080abf8ed443db1fd04f351d7ed6f (table test-table [id=c85e6f9fd9df4e52af9c79089e7ba997]) successfully deleted
I20260619 14:41:16.808796 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 17777
I20260619 14:41:16.813644 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 17415
2026-06-19T14:41:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2208 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-19T14:41:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:41:16Z Disabled control of system clock
I20260619 14:41:16.841333 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36323
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36323
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 14:41:16.949098 17952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:41:16.949373 17952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:41:16.949407 17952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:41:16.953348 17952 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:41:16.953408 17952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:41:16.953421 17952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:41:16.953437 17952 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:41:16.953451 17952 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:41:16.958236 17952 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36323
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:41:16.959295 17952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:41:16.960604 17952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:41:16.969123 17958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:16.969138 17957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:16.969174 17960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:16.969467 17952 server_base.cc:1034] running on GCE node
I20260619 14:41:16.970077 17952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:41:16.971210 17952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:41:16.972378 17952 hybrid_clock.cc:648] HybridClock initialized: now 1781880076972361 us; error 32 us; skew 500 ppm
I20260619 14:41:16.974408 17952 webserver.cc:460] Webserver started at http://127.0.98.62:35339/ using document root <none> and password file <none>
I20260619 14:41:16.974987 17952 fs_manager.cc:359] Metadata directory not provided
I20260619 14:41:16.975050 17952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:41:16.975246 17952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:41:16.977423 17952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/master-0/data/instance:
uuid: "3fec07b64bd64536b4efac69fa80907b"
format_stamp: "Formatted at 2026-06-19 14:41:16 on dist-test-slave-32hj"
server_key: "d2e3b270132927c3cc71794f000397cd"
server_key_iv: "7165bb1029a5a21e3b3d66bb3e9cf960"
server_key_version: "encryptionkey@0"
I20260619 14:41:16.977969 17952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/master-0/wal/instance:
uuid: "3fec07b64bd64536b4efac69fa80907b"
format_stamp: "Formatted at 2026-06-19 14:41:16 on dist-test-slave-32hj"
server_key: "d2e3b270132927c3cc71794f000397cd"
server_key_iv: "7165bb1029a5a21e3b3d66bb3e9cf960"
server_key_version: "encryptionkey@0"
I20260619 14:41:16.981685 17952 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 14:41:16.984160 17966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:16.985510 17952 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 14:41:16.985631 17952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/master-0/wal
uuid: "3fec07b64bd64536b4efac69fa80907b"
format_stamp: "Formatted at 2026-06-19 14:41:16 on dist-test-slave-32hj"
server_key: "d2e3b270132927c3cc71794f000397cd"
server_key_iv: "7165bb1029a5a21e3b3d66bb3e9cf960"
server_key_version: "encryptionkey@0"
I20260619 14:41:16.985733 17952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:41:17.012251 17952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:41:17.013053 17952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:41:17.013232 17952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:41:17.020076 17952 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36323
I20260619 14:41:17.020094 18018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36323 every 8 connection(s)
I20260619 14:41:17.021088 17952 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:41:17.024209 18019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:41:17.026953 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 17952
I20260619 14:41:17.027073 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/master-0/wal/instance
I20260619 14:41:17.027330 392 external_mini_cluster.cc:1428] Setting key f8c9985a39030de9e65b53652a29bde7
I20260619 14:41:17.030484 18019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fec07b64bd64536b4efac69fa80907b: Bootstrap starting.
I20260619 14:41:17.033301 18019 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3fec07b64bd64536b4efac69fa80907b: Neither blocks nor log segments found. Creating new log.
I20260619 14:41:17.034274 18019 log.cc:824] T 00000000000000000000000000000000 P 3fec07b64bd64536b4efac69fa80907b: Log is configured to *not* fsync() on all Append() calls
I20260619 14:41:17.036666 18019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fec07b64bd64536b4efac69fa80907b: No bootstrap required, opened a new log
I20260619 14:41:17.038157 18022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:17.028857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46952 (local address 127.0.98.62:36323)
0619 14:41:17.029569 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:17.029591 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:17.029636 (+ 45us) server_negotiation.cc:407] Connection header received
0619 14:41:17.030447 (+ 811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:17.030470 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:17.030999 (+ 529us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:17.031374 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:17.031859 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:17.033317 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:17.034611 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:17.035287 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:17.035433 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:17.036308 (+ 875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:17.036313 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:17.036328 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:41:17.036770 (+ 442us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:17.036844 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:41:17.036872 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:17.036969 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:17.037118 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 14:41:17.037394 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":110,"threads_started":1}
I20260619 14:41:17.040679 18019 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3fec07b64bd64536b4efac69fa80907b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fec07b64bd64536b4efac69fa80907b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36323 } }
I20260619 14:41:17.040946 18019 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3fec07b64bd64536b4efac69fa80907b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:41:17.041013 18019 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3fec07b64bd64536b4efac69fa80907b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fec07b64bd64536b4efac69fa80907b, State: Initialized, Role: FOLLOWER
I20260619 14:41:17.041440 18019 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3fec07b64bd64536b4efac69fa80907b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fec07b64bd64536b4efac69fa80907b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36323 } }
I20260619 14:41:17.041574 18019 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3fec07b64bd64536b4efac69fa80907b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:41:17.041642 18019 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3fec07b64bd64536b4efac69fa80907b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:41:17.041736 18019 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3fec07b64bd64536b4efac69fa80907b [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:41:17.042802 18019 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3fec07b64bd64536b4efac69fa80907b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fec07b64bd64536b4efac69fa80907b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36323 } }
I20260619 14:41:17.043102 18019 leader_election.cc:302] T 00000000000000000000000000000000 P 3fec07b64bd64536b4efac69fa80907b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3fec07b64bd64536b4efac69fa80907b; no voters:
I20260619 14:41:17.043497 18019 leader_election.cc:288] T 00000000000000000000000000000000 P 3fec07b64bd64536b4efac69fa80907b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:41:17.043573 18024 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3fec07b64bd64536b4efac69fa80907b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:41:17.043808 18024 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3fec07b64bd64536b4efac69fa80907b [term 1 LEADER]: Becoming Leader. State: Replica: 3fec07b64bd64536b4efac69fa80907b, State: Running, Role: LEADER
I20260619 14:41:17.044128 18024 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3fec07b64bd64536b4efac69fa80907b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fec07b64bd64536b4efac69fa80907b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36323 } }
I20260619 14:41:17.044682 18019 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3fec07b64bd64536b4efac69fa80907b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:41:17.045694 18026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fec07b64bd64536b4efac69fa80907b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3fec07b64bd64536b4efac69fa80907b. Latest consensus state: current_term: 1 leader_uuid: "3fec07b64bd64536b4efac69fa80907b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fec07b64bd64536b4efac69fa80907b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36323 } } }
I20260619 14:41:17.045819 18026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fec07b64bd64536b4efac69fa80907b [sys.catalog]: This master's current role is: LEADER
I20260619 14:41:17.046072 18025 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fec07b64bd64536b4efac69fa80907b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3fec07b64bd64536b4efac69fa80907b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fec07b64bd64536b4efac69fa80907b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36323 } } }
I20260619 14:41:17.046156 18025 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fec07b64bd64536b4efac69fa80907b [sys.catalog]: This master's current role is: LEADER
W20260619 14:41:17.049450 18040 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3fec07b64bd64536b4efac69fa80907b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:41:17.049521 18040 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:41:17.049590 18033 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:41:17.050257 18033 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:41:17.055604 18033 catalog_manager.cc:1347] Generated new cluster ID: b090677ab73c4bf9885347674ff59a13
I20260619 14:41:17.055673 18033 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:41:17.063752 18033 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:41:17.064628 18033 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:41:17.073553 18033 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3fec07b64bd64536b4efac69fa80907b: Generated new TSK 0
I20260619 14:41:17.074301 18033 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:41:17.079128 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36323
--builtin_ntp_servers=127.0.98.20:39791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 14:41:17.188656 18043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:41:17.188930 18043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:41:17.188969 18043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:41:17.193005 18043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:41:17.193063 18043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:41:17.193120 18043 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:41:17.197831 18043 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18043
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:41:17.198866 18043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:41:17.200196 18043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:41:17.209534 18048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:17.209546 18049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:17.209510 18051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:17.209789 18043 server_base.cc:1034] running on GCE node
I20260619 14:41:17.210285 18043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:41:17.210940 18043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:41:17.212116 18043 hybrid_clock.cc:648] HybridClock initialized: now 1781880077212089 us; error 36 us; skew 500 ppm
I20260619 14:41:17.220206 18043 webserver.cc:460] Webserver started at http://127.0.98.1:41371/ using document root <none> and password file <none>
I20260619 14:41:17.220868 18043 fs_manager.cc:359] Metadata directory not provided
I20260619 14:41:17.220950 18043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:41:17.221158 18043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:41:17.223227 18043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/ts-0/data/instance:
uuid: "e0e2af5a4ac440e8b40a9a0686eb5806"
format_stamp: "Formatted at 2026-06-19 14:41:17 on dist-test-slave-32hj"
server_key: "b141af9b7bc5cb255a11482c1626f59b"
server_key_iv: "4d0b660c0d0d1bed881a5a7d931aaf3d"
server_key_version: "encryptionkey@0"
I20260619 14:41:17.223752 18043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e0e2af5a4ac440e8b40a9a0686eb5806"
format_stamp: "Formatted at 2026-06-19 14:41:17 on dist-test-slave-32hj"
server_key: "b141af9b7bc5cb255a11482c1626f59b"
server_key_iv: "4d0b660c0d0d1bed881a5a7d931aaf3d"
server_key_version: "encryptionkey@0"
I20260619 14:41:17.227485 18043 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260619 14:41:17.229996 18057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:17.231243 18043 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:41:17.231386 18043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/ts-0/wal
uuid: "e0e2af5a4ac440e8b40a9a0686eb5806"
format_stamp: "Formatted at 2026-06-19 14:41:17 on dist-test-slave-32hj"
server_key: "b141af9b7bc5cb255a11482c1626f59b"
server_key_iv: "4d0b660c0d0d1bed881a5a7d931aaf3d"
server_key_version: "encryptionkey@0"
I20260619 14:41:17.231499 18043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:41:17.250074 18043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:41:17.250808 18043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:41:17.250993 18043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:41:17.251582 18043 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:41:17.252513 18043 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:41:17.252559 18043 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:17.252611 18043 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:41:17.252640 18043 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:17.260919 18043 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36049
I20260619 14:41:17.260959 18170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36049 every 8 connection(s)
I20260619 14:41:17.261833 18043 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:41:17.265297 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 18043
I20260619 14:41:17.265424 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:41:17.265682 392 external_mini_cluster.cc:1428] Setting key 9b6b85b151efe10f703b62063c0cdfb1
I20260619 14:41:17.267504 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36323
--builtin_ntp_servers=127.0.98.20:39791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 14:41:17.271579 18022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:17.263644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46345 (local address 127.0.98.62:36323)
0619 14:41:17.263779 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:17.263784 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:17.264942 (+ 1158us) server_negotiation.cc:407] Connection header received
0619 14:41:17.266465 (+ 1523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:17.266468 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:17.266521 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:17.266595 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:17.267568 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:17.268153 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:17.269759 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:17.269993 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:17.270082 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:17.271136 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:17.271139 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:17.271141 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:41:17.271153 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:17.271171 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:41:17.271179 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:17.271235 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:17.271379 (+ 144us) server_negotiation.cc:299] Negotiation successful
0619 14:41:17.271439 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 14:41:17.272536 18173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:17.264080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36323 (local address 127.0.98.1:46345)
0619 14:41:17.264795 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:17.264839 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:17.266229 (+ 1390us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:17.266792 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:17.266801 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:17.266917 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:41:17.267431 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:17.267447 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:17.268280 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:17.268283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:17.269646 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:17.269657 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:17.270136 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:17.270140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:17.270315 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:17.270887 (+ 572us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:41:17.270888 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:41:17.270930 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:41:17.270996 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:41:17.271244 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:41:17.271253 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:41:17.271258 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 14:41:17.271372 (+ 114us) client_negotiation.cc:240] Negotiation successful
0619 14:41:17.271665 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":105,"threads_started":1}
I20260619 14:41:17.273720 18171 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36323
I20260619 14:41:17.273977 18171 heartbeater.cc:464] Registering TS with master...
I20260619 14:41:17.274608 18171 heartbeater.cc:511] Master 127.0.98.62:36323 requested a full tablet report, sending...
I20260619 14:41:17.276398 17982 ts_manager.cc:194] Registered new tserver with Master: e0e2af5a4ac440e8b40a9a0686eb5806 (127.0.98.1:36049)
I20260619 14:41:17.278005 17982 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:46345
W20260619 14:41:17.379343 18174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:41:17.379602 18174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:41:17.379639 18174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:41:17.383591 18174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:41:17.383641 18174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:41:17.383692 18174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:41:17.388300 18174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:41:17.389331 18174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:41:17.390663 18174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:41:17.399875 18179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:17.399958 18174 server_base.cc:1034] running on GCE node
W20260619 14:41:17.399919 18180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:17.399906 18182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:17.400528 18174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:41:17.401180 18174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:41:17.402365 18174 hybrid_clock.cc:648] HybridClock initialized: now 1781880077402334 us; error 42 us; skew 500 ppm
I20260619 14:41:17.404181 18174 webserver.cc:460] Webserver started at http://127.0.98.2:38409/ using document root <none> and password file <none>
I20260619 14:41:17.404762 18174 fs_manager.cc:359] Metadata directory not provided
I20260619 14:41:17.404829 18174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:41:17.405030 18174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:41:17.407145 18174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/ts-1/data/instance:
uuid: "c3fa4f56c1e1420c98b0979ec6066a9f"
format_stamp: "Formatted at 2026-06-19 14:41:17 on dist-test-slave-32hj"
server_key: "7eb0c21a1db860e03d1076fc2aab08c8"
server_key_iv: "8ac00599334e5109624fd85cb760e7b5"
server_key_version: "encryptionkey@0"
I20260619 14:41:17.407661 18174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c3fa4f56c1e1420c98b0979ec6066a9f"
format_stamp: "Formatted at 2026-06-19 14:41:17 on dist-test-slave-32hj"
server_key: "7eb0c21a1db860e03d1076fc2aab08c8"
server_key_iv: "8ac00599334e5109624fd85cb760e7b5"
server_key_version: "encryptionkey@0"
I20260619 14:41:17.411134 18174 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:41:17.413522 18188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:17.414712 18174 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:41:17.414812 18174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/ts-1/wal
uuid: "c3fa4f56c1e1420c98b0979ec6066a9f"
format_stamp: "Formatted at 2026-06-19 14:41:17 on dist-test-slave-32hj"
server_key: "7eb0c21a1db860e03d1076fc2aab08c8"
server_key_iv: "8ac00599334e5109624fd85cb760e7b5"
server_key_version: "encryptionkey@0"
I20260619 14:41:17.414911 18174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:41:17.429279 18174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:41:17.429946 18174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:41:17.430109 18174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:41:17.430673 18174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:41:17.431579 18174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:41:17.431630 18174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:17.431674 18174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:41:17.431702 18174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:17.440155 18174 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35495
I20260619 14:41:17.440186 18301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35495 every 8 connection(s)
I20260619 14:41:17.441051 18174 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:41:17.443485 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 18174
I20260619 14:41:17.443600 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:41:17.443859 392 external_mini_cluster.cc:1428] Setting key 549ae83037924aca173a5cd6008122e2
I20260619 14:41:17.445796 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36323
--builtin_ntp_servers=127.0.98.20:39791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 14:41:17.450122 18022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:17.442700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43787 (local address 127.0.98.62:36323)
0619 14:41:17.442855 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:17.442860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:17.443948 (+ 1088us) server_negotiation.cc:407] Connection header received
0619 14:41:17.445571 (+ 1623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:17.445575 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:17.445626 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:17.445692 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:17.446627 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:17.447148 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:17.448283 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:17.448499 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:17.448585 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:17.449665 (+ 1080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:17.449668 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:17.449670 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:41:17.449682 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:17.449700 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:41:17.449708 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:17.449772 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:17.449915 (+ 143us) server_negotiation.cc:299] Negotiation successful
0619 14:41:17.449978 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 14:41:17.450989 18304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:17.443110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36323 (local address 127.0.98.2:43787)
0619 14:41:17.443799 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:17.443849 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:17.445352 (+ 1503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:17.445862 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:17.445869 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:17.445975 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:41:17.446498 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:17.446511 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:17.447277 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:17.447279 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:17.448163 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:17.448172 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:17.448644 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:17.448648 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:17.448843 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:17.449413 (+ 570us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:41:17.449415 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:41:17.449456 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:41:17.449519 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:41:17.449782 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:41:17.449791 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:41:17.449796 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 14:41:17.449905 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 14:41:17.450193 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":117,"threads_started":1}
I20260619 14:41:17.452096 18302 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36323
I20260619 14:41:17.452369 18302 heartbeater.cc:464] Registering TS with master...
I20260619 14:41:17.453060 18302 heartbeater.cc:511] Master 127.0.98.62:36323 requested a full tablet report, sending...
I20260619 14:41:17.454212 17982 ts_manager.cc:194] Registered new tserver with Master: c3fa4f56c1e1420c98b0979ec6066a9f (127.0.98.2:35495)
I20260619 14:41:17.454990 17982 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:43787
W20260619 14:41:17.555266 18305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:41:17.555553 18305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:41:17.555591 18305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:41:17.559592 18305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:41:17.559660 18305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:41:17.559716 18305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:41:17.564443 18305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:41:17.565486 18305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:41:17.566771 18305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:41:17.576431 18313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:17.576570 18305 server_base.cc:1034] running on GCE node
W20260619 14:41:17.576428 18310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:17.576454 18311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:17.577190 18305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:41:17.577841 18305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:41:17.579020 18305 hybrid_clock.cc:648] HybridClock initialized: now 1781880077579004 us; error 27 us; skew 500 ppm
I20260619 14:41:17.580806 18305 webserver.cc:460] Webserver started at http://127.0.98.3:40813/ using document root <none> and password file <none>
I20260619 14:41:17.581358 18305 fs_manager.cc:359] Metadata directory not provided
I20260619 14:41:17.581427 18305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:41:17.581590 18305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:41:17.583536 18305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/ts-2/data/instance:
uuid: "912eb9fd34324a53a521635acb492be0"
format_stamp: "Formatted at 2026-06-19 14:41:17 on dist-test-slave-32hj"
server_key: "01fca1e903252ef0fcd8c373c792828f"
server_key_iv: "6efd14329412783e81d84470af07ad83"
server_key_version: "encryptionkey@0"
I20260619 14:41:17.584008 18305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/ts-2/wal/instance:
uuid: "912eb9fd34324a53a521635acb492be0"
format_stamp: "Formatted at 2026-06-19 14:41:17 on dist-test-slave-32hj"
server_key: "01fca1e903252ef0fcd8c373c792828f"
server_key_iv: "6efd14329412783e81d84470af07ad83"
server_key_version: "encryptionkey@0"
I20260619 14:41:17.587463 18305 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 14:41:17.589740 18319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:17.590941 18305 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:41:17.591046 18305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/ts-2/wal
uuid: "912eb9fd34324a53a521635acb492be0"
format_stamp: "Formatted at 2026-06-19 14:41:17 on dist-test-slave-32hj"
server_key: "01fca1e903252ef0fcd8c373c792828f"
server_key_iv: "6efd14329412783e81d84470af07ad83"
server_key_version: "encryptionkey@0"
I20260619 14:41:17.591151 18305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:41:17.604557 18305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:41:17.605175 18305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:41:17.605306 18305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:41:17.605786 18305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:41:17.606626 18305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:41:17.606669 18305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:17.606705 18305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:41:17.606724 18305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:17.614833 18305 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37239
I20260619 14:41:17.614907 18432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37239 every 8 connection(s)
I20260619 14:41:17.615693 18305 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:41:17.621796 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 18305
I20260619 14:41:17.621918 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:41:17.622186 392 external_mini_cluster.cc:1428] Setting key 2bd68bc3290f04dad6f2e959edb8a8a5
I20260619 14:41:17.624816 18022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:17.617354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57983 (local address 127.0.98.62:36323)
0619 14:41:17.617523 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:17.617527 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:17.618577 (+ 1050us) server_negotiation.cc:407] Connection header received
0619 14:41:17.619860 (+ 1283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:17.619863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:17.619911 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:17.619985 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:17.620848 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:17.621366 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:17.622788 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:17.623094 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:17.623173 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:17.624342 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:17.624345 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:17.624347 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:41:17.624358 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:17.624375 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:41:17.624383 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:17.624457 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:17.624613 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 14:41:17.624665 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 14:41:17.625592 18435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:17.617779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36323 (local address 127.0.98.3:57983)
0619 14:41:17.618432 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:17.618472 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:17.619676 (+ 1204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:17.620126 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:17.620132 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:17.620213 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:41:17.620697 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:17.620709 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:17.621472 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:17.621474 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:17.622683 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:17.622692 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:17.623232 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:17.623237 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:17.623446 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:17.624086 (+ 640us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:41:17.624089 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:41:17.624135 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:41:17.624211 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:41:17.624465 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:41:17.624470 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:41:17.624474 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 14:41:17.624559 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 14:41:17.624812 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":134,"threads_started":1}
I20260619 14:41:17.626579 18433 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36323
I20260619 14:41:17.626781 18433 heartbeater.cc:464] Registering TS with master...
I20260619 14:41:17.627257 18433 heartbeater.cc:511] Master 127.0.98.62:36323 requested a full tablet report, sending...
I20260619 14:41:17.628163 17982 ts_manager.cc:194] Registered new tserver with Master: 912eb9fd34324a53a521635acb492be0 (127.0.98.3:37239)
I20260619 14:41:17.628698 17982 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:57983
I20260619 14:41:17.635648 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:41:17.640818 18022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:17.637514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46960 (local address 127.0.98.62:36323)
0619 14:41:17.637632 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:17.637635 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:17.637805 (+ 170us) server_negotiation.cc:407] Connection header received
0619 14:41:17.637897 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:17.637900 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:17.637935 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:17.637999 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:17.638204 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:17.638723 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:17.639544 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:17.639741 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:17.639813 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:17.640486 (+ 673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:17.640488 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:17.640490 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:41:17.640499 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:17.640512 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:41:17.640518 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:17.640565 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:17.640631 (+ 66us) server_negotiation.cc:299] Negotiation successful
0619 14:41:17.640674 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 14:41:17.643328 17982 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46960:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:41:17.646313 17982 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:41:17.659652 18447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:17.654186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37239 (local address 127.0.0.1:41350)
0619 14:41:17.654747 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:17.654757 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:17.654820 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:17.655753 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:17.655759 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:17.655780 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:41:17.655894 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:17.655901 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:17.657113 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:17.657117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:17.658290 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:17.658299 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:17.659089 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:17.659092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:17.659241 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:17.659283 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 14:41:17.659392 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 14:41:17.659502 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"mutex_wait_us":20,"thread_start_us":97,"threads_started":1}
I20260619 14:41:17.659698 18449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:17.655007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41350 (local address 127.0.98.3:37239)
0619 14:41:17.655323 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:17.655334 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:17.655358 (+ 24us) server_negotiation.cc:407] Connection header received
0619 14:41:17.655434 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:17.655439 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:17.655612 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:17.655747 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:41:17.656016 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:17.657004 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:17.658405 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:17.658983 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:17.659092 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:17.659275 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:17.659438 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 14:41:17.659535 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":134,"threads_started":1}
I20260619 14:41:17.660331 18444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:17.653933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36049 (local address 127.0.0.1:37054)
0619 14:41:17.654347 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:17.654405 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:17.654520 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:17.656303 (+ 1783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:17.656306 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:17.656322 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:41:17.656434 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:17.656441 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:17.657590 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:17.657594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:17.658722 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:17.658732 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:17.659942 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:17.659945 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:17.660046 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:17.660069 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 14:41:17.660144 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 14:41:17.660210 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":62,"thread_start_us":70,"threads_started":1}
I20260619 14:41:17.660671 18445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:17.654115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37054 (local address 127.0.98.1:36049)
0619 14:41:17.655778 (+ 1663us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:17.655787 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:17.655808 (+ 21us) server_negotiation.cc:407] Connection header received
0619 14:41:17.655874 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:17.655879 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:17.656033 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:17.656161 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:41:17.656553 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:17.657478 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:17.658852 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:17.659591 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:17.659689 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:17.659865 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:17.660195 (+ 330us) server_negotiation.cc:299] Negotiation successful
0619 14:41:17.660293 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1565,"thread_start_us":96,"threads_started":1}
I20260619 14:41:17.661716 18446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:17.654552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35495 (local address 127.0.0.1:59798)
0619 14:41:17.655423 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:17.655436 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:17.655532 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:17.655947 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:17.655950 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:17.655966 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:41:17.656053 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:17.656060 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:17.658426 (+ 2366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:17.658429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:17.659498 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:17.659507 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:17.660682 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:17.660686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:17.660800 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:17.660821 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 14:41:17.660883 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 14:41:17.661599 (+ 716us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":802,"mutex_wait_us":33,"thread_start_us":85,"threads_started":1}
I20260619 14:41:17.662218 18448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:17.654730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59798 (local address 127.0.98.2:35495)
0619 14:41:17.655073 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:17.655086 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:17.655533 (+ 447us) server_negotiation.cc:407] Connection header received
0619 14:41:17.655638 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:17.655643 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:17.655824 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:17.655977 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:41:17.656141 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:17.657014 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:17.659621 (+ 2607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:17.660346 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:17.660457 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:17.660624 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:17.661974 (+ 1350us) server_negotiation.cc:299] Negotiation successful
0619 14:41:17.662072 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":161,"threads_started":1}
I20260619 14:41:17.662756 18367 tablet_service.cc:1467] Processing CreateTablet for tablet c4303f5676d74e3d9cb57b3d4fbff86a (DEFAULT_TABLE table=test-table [id=ee6eed2c5d6a4cc1815a0c2b5c54ccaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:41:17.663467 18105 tablet_service.cc:1467] Processing CreateTablet for tablet c4303f5676d74e3d9cb57b3d4fbff86a (DEFAULT_TABLE table=test-table [id=ee6eed2c5d6a4cc1815a0c2b5c54ccaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:41:17.663846 18367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4303f5676d74e3d9cb57b3d4fbff86a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:41:17.664278 18105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4303f5676d74e3d9cb57b3d4fbff86a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:41:17.664798 18236 tablet_service.cc:1467] Processing CreateTablet for tablet c4303f5676d74e3d9cb57b3d4fbff86a (DEFAULT_TABLE table=test-table [id=ee6eed2c5d6a4cc1815a0c2b5c54ccaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:41:17.665742 18236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4303f5676d74e3d9cb57b3d4fbff86a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:41:17.670203 18450 tablet_bootstrap.cc:492] T c4303f5676d74e3d9cb57b3d4fbff86a P e0e2af5a4ac440e8b40a9a0686eb5806: Bootstrap starting.
I20260619 14:41:17.671507 18451 tablet_bootstrap.cc:492] T c4303f5676d74e3d9cb57b3d4fbff86a P 912eb9fd34324a53a521635acb492be0: Bootstrap starting.
I20260619 14:41:17.672103 18452 tablet_bootstrap.cc:492] T c4303f5676d74e3d9cb57b3d4fbff86a P c3fa4f56c1e1420c98b0979ec6066a9f: Bootstrap starting.
I20260619 14:41:17.672329 18450 tablet_bootstrap.cc:654] T c4303f5676d74e3d9cb57b3d4fbff86a P e0e2af5a4ac440e8b40a9a0686eb5806: Neither blocks nor log segments found. Creating new log.
I20260619 14:41:17.673202 18450 log.cc:824] T c4303f5676d74e3d9cb57b3d4fbff86a P e0e2af5a4ac440e8b40a9a0686eb5806: Log is configured to *not* fsync() on all Append() calls
I20260619 14:41:17.673434 18451 tablet_bootstrap.cc:654] T c4303f5676d74e3d9cb57b3d4fbff86a P 912eb9fd34324a53a521635acb492be0: Neither blocks nor log segments found. Creating new log.
I20260619 14:41:17.674085 18452 tablet_bootstrap.cc:654] T c4303f5676d74e3d9cb57b3d4fbff86a P c3fa4f56c1e1420c98b0979ec6066a9f: Neither blocks nor log segments found. Creating new log.
I20260619 14:41:17.674155 18451 log.cc:824] T c4303f5676d74e3d9cb57b3d4fbff86a P 912eb9fd34324a53a521635acb492be0: Log is configured to *not* fsync() on all Append() calls
I20260619 14:41:17.674798 18452 log.cc:824] T c4303f5676d74e3d9cb57b3d4fbff86a P c3fa4f56c1e1420c98b0979ec6066a9f: Log is configured to *not* fsync() on all Append() calls
I20260619 14:41:17.675386 18450 tablet_bootstrap.cc:492] T c4303f5676d74e3d9cb57b3d4fbff86a P e0e2af5a4ac440e8b40a9a0686eb5806: No bootstrap required, opened a new log
I20260619 14:41:17.675575 18450 ts_tablet_manager.cc:1397] T c4303f5676d74e3d9cb57b3d4fbff86a P e0e2af5a4ac440e8b40a9a0686eb5806: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 14:41:17.675958 18451 tablet_bootstrap.cc:492] T c4303f5676d74e3d9cb57b3d4fbff86a P 912eb9fd34324a53a521635acb492be0: No bootstrap required, opened a new log
I20260619 14:41:17.676116 18451 ts_tablet_manager.cc:1397] T c4303f5676d74e3d9cb57b3d4fbff86a P 912eb9fd34324a53a521635acb492be0: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 14:41:17.678282 18452 tablet_bootstrap.cc:492] T c4303f5676d74e3d9cb57b3d4fbff86a P c3fa4f56c1e1420c98b0979ec6066a9f: No bootstrap required, opened a new log
I20260619 14:41:17.678458 18452 ts_tablet_manager.cc:1397] T c4303f5676d74e3d9cb57b3d4fbff86a P c3fa4f56c1e1420c98b0979ec6066a9f: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260619 14:41:17.679243 18451 raft_consensus.cc:348] T c4303f5676d74e3d9cb57b3d4fbff86a P 912eb9fd34324a53a521635acb492be0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e2af5a4ac440e8b40a9a0686eb5806" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36049 } } peers { permanent_uuid: "912eb9fd34324a53a521635acb492be0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37239 } } peers { permanent_uuid: "c3fa4f56c1e1420c98b0979ec6066a9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35495 } }
I20260619 14:41:17.679456 18451 raft_consensus.cc:374] T c4303f5676d74e3d9cb57b3d4fbff86a P 912eb9fd34324a53a521635acb492be0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:41:17.679298 18450 raft_consensus.cc:348] T c4303f5676d74e3d9cb57b3d4fbff86a P e0e2af5a4ac440e8b40a9a0686eb5806 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e2af5a4ac440e8b40a9a0686eb5806" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36049 } } peers { permanent_uuid: "912eb9fd34324a53a521635acb492be0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37239 } } peers { permanent_uuid: "c3fa4f56c1e1420c98b0979ec6066a9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35495 } }
I20260619 14:41:17.679505 18451 raft_consensus.cc:729] T c4303f5676d74e3d9cb57b3d4fbff86a P 912eb9fd34324a53a521635acb492be0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 912eb9fd34324a53a521635acb492be0, State: Initialized, Role: FOLLOWER
I20260619 14:41:17.679546 18450 raft_consensus.cc:374] T c4303f5676d74e3d9cb57b3d4fbff86a P e0e2af5a4ac440e8b40a9a0686eb5806 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:41:17.679605 18450 raft_consensus.cc:729] T c4303f5676d74e3d9cb57b3d4fbff86a P e0e2af5a4ac440e8b40a9a0686eb5806 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0e2af5a4ac440e8b40a9a0686eb5806, State: Initialized, Role: FOLLOWER
I20260619 14:41:17.679926 18451 consensus_queue.cc:260] T c4303f5676d74e3d9cb57b3d4fbff86a P 912eb9fd34324a53a521635acb492be0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e2af5a4ac440e8b40a9a0686eb5806" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36049 } } peers { permanent_uuid: "912eb9fd34324a53a521635acb492be0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37239 } } peers { permanent_uuid: "c3fa4f56c1e1420c98b0979ec6066a9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35495 } }
I20260619 14:41:17.680003 18450 consensus_queue.cc:260] T c4303f5676d74e3d9cb57b3d4fbff86a P e0e2af5a4ac440e8b40a9a0686eb5806 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e2af5a4ac440e8b40a9a0686eb5806" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36049 } } peers { permanent_uuid: "912eb9fd34324a53a521635acb492be0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37239 } } peers { permanent_uuid: "c3fa4f56c1e1420c98b0979ec6066a9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35495 } }
I20260619 14:41:17.680507 18433 heartbeater.cc:503] Master 127.0.98.62:36323 was elected leader, sending a full tablet report...
I20260619 14:41:17.680864 18451 ts_tablet_manager.cc:1428] T c4303f5676d74e3d9cb57b3d4fbff86a P 912eb9fd34324a53a521635acb492be0: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 14:41:17.680912 18450 ts_tablet_manager.cc:1428] T c4303f5676d74e3d9cb57b3d4fbff86a P e0e2af5a4ac440e8b40a9a0686eb5806: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 14:41:17.681283 18171 heartbeater.cc:503] Master 127.0.98.62:36323 was elected leader, sending a full tablet report...
I20260619 14:41:17.683985 18452 raft_consensus.cc:348] T c4303f5676d74e3d9cb57b3d4fbff86a P c3fa4f56c1e1420c98b0979ec6066a9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e2af5a4ac440e8b40a9a0686eb5806" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36049 } } peers { permanent_uuid: "912eb9fd34324a53a521635acb492be0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37239 } } peers { permanent_uuid: "c3fa4f56c1e1420c98b0979ec6066a9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35495 } }
I20260619 14:41:17.684193 18452 raft_consensus.cc:374] T c4303f5676d74e3d9cb57b3d4fbff86a P c3fa4f56c1e1420c98b0979ec6066a9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:41:17.684247 18452 raft_consensus.cc:729] T c4303f5676d74e3d9cb57b3d4fbff86a P c3fa4f56c1e1420c98b0979ec6066a9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3fa4f56c1e1420c98b0979ec6066a9f, State: Initialized, Role: FOLLOWER
I20260619 14:41:17.684631 18452 consensus_queue.cc:260] T c4303f5676d74e3d9cb57b3d4fbff86a P c3fa4f56c1e1420c98b0979ec6066a9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e2af5a4ac440e8b40a9a0686eb5806" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36049 } } peers { permanent_uuid: "912eb9fd34324a53a521635acb492be0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37239 } } peers { permanent_uuid: "c3fa4f56c1e1420c98b0979ec6066a9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35495 } }
I20260619 14:41:17.685535 18302 heartbeater.cc:503] Master 127.0.98.62:36323 was elected leader, sending a full tablet report...
I20260619 14:41:17.685614 18452 ts_tablet_manager.cc:1428] T c4303f5676d74e3d9cb57b3d4fbff86a P c3fa4f56c1e1420c98b0979ec6066a9f: Time spent starting tablet: real 0.007s user 0.001s sys 0.005s
W20260619 14:41:17.691843 18303 tablet.cc:2369] T c4303f5676d74e3d9cb57b3d4fbff86a P c3fa4f56c1e1420c98b0979ec6066a9f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:41:17.718443 18456 raft_consensus.cc:482] T c4303f5676d74e3d9cb57b3d4fbff86a P 912eb9fd34324a53a521635acb492be0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:41:17.718659 18456 raft_consensus.cc:504] T c4303f5676d74e3d9cb57b3d4fbff86a P 912eb9fd34324a53a521635acb492be0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e2af5a4ac440e8b40a9a0686eb5806" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36049 } } peers { permanent_uuid: "912eb9fd34324a53a521635acb492be0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37239 } } peers { permanent_uuid: "c3fa4f56c1e1420c98b0979ec6066a9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35495 } }
I20260619 14:41:17.719808 18456 leader_election.cc:288] T c4303f5676d74e3d9cb57b3d4fbff86a P 912eb9fd34324a53a521635acb492be0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0e2af5a4ac440e8b40a9a0686eb5806 (127.0.98.1:36049), c3fa4f56c1e1420c98b0979ec6066a9f (127.0.98.2:35495)
I20260619 14:41:17.724499 18445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:17.720166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42981 (local address 127.0.98.1:36049)
0619 14:41:17.720321 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:17.720326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:17.720343 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:41:17.720409 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:17.720413 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:17.720463 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:17.720529 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:41:17.720850 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:17.721662 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:17.723274 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:17.723876 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:17.723983 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:17.724044 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:17.724292 (+ 248us) server_negotiation.cc:299] Negotiation successful
0619 14:41:17.724366 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 14:41:17.724499 18448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:17.720289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46625 (local address 127.0.98.2:35495)
0619 14:41:17.720401 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:17.720405 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:17.720585 (+ 180us) server_negotiation.cc:407] Connection header received
0619 14:41:17.720708 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:17.720710 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:17.720779 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:17.720852 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:41:17.721067 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:17.721751 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:17.723275 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:17.723871 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:17.723983 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:17.724044 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:17.724292 (+ 248us) server_negotiation.cc:299] Negotiation successful
0619 14:41:17.724363 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 14:41:17.724539 18435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:17.720059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36049 (local address 127.0.98.3:42981)
0619 14:41:17.720170 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:17.720185 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:17.720288 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:17.720572 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:17.720575 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:17.720599 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:41:17.720749 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:17.720756 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:17.721768 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:17.721770 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:17.723145 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:17.723157 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:17.724005 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:17.724009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:17.724159 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:17.724198 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 14:41:17.724267 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 14:41:17.724381 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":153}
I20260619 14:41:17.724555 18459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:17.720061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35495 (local address 127.0.98.3:46625)
0619 14:41:17.720489 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:17.720500 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:17.720620 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:17.720872 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:17.720874 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:17.720889 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:41:17.720972 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:17.720976 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:17.721836 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:17.721839 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:17.723145 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:17.723157 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:17.724005 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:17.724009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:17.724155 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:17.724193 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 14:41:17.724267 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 14:41:17.724376 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"mutex_wait_us":115,"spinlock_wait_cycles":50560,"thread_start_us":123,"threads_started":1}
I20260619 14:41:17.725239 18125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4303f5676d74e3d9cb57b3d4fbff86a" candidate_uuid: "912eb9fd34324a53a521635acb492be0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0e2af5a4ac440e8b40a9a0686eb5806" is_pre_election: true
I20260619 14:41:17.725394 18256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4303f5676d74e3d9cb57b3d4fbff86a" candidate_uuid: "912eb9fd34324a53a521635acb492be0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3fa4f56c1e1420c98b0979ec6066a9f" is_pre_election: true
I20260619 14:41:17.725621 18256 raft_consensus.cc:2454] T c4303f5676d74e3d9cb57b3d4fbff86a P c3fa4f56c1e1420c98b0979ec6066a9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 912eb9fd34324a53a521635acb492be0 in term 0.
I20260619 14:41:17.725621 18125 raft_consensus.cc:2454] T c4303f5676d74e3d9cb57b3d4fbff86a P e0e2af5a4ac440e8b40a9a0686eb5806 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 912eb9fd34324a53a521635acb492be0 in term 0.
I20260619 14:41:17.726145 18322 leader_election.cc:302] T c4303f5676d74e3d9cb57b3d4fbff86a P 912eb9fd34324a53a521635acb492be0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 912eb9fd34324a53a521635acb492be0, c3fa4f56c1e1420c98b0979ec6066a9f; no voters:
I20260619 14:41:17.726437 18456 raft_consensus.cc:2781] T c4303f5676d74e3d9cb57b3d4fbff86a P 912eb9fd34324a53a521635acb492be0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:41:17.726522 18456 raft_consensus.cc:482] T c4303f5676d74e3d9cb57b3d4fbff86a P 912eb9fd34324a53a521635acb492be0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:41:17.726572 18456 raft_consensus.cc:3037] T c4303f5676d74e3d9cb57b3d4fbff86a P 912eb9fd34324a53a521635acb492be0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:41:17.727550 18456 raft_consensus.cc:504] T c4303f5676d74e3d9cb57b3d4fbff86a P 912eb9fd34324a53a521635acb492be0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e2af5a4ac440e8b40a9a0686eb5806" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36049 } } peers { permanent_uuid: "912eb9fd34324a53a521635acb492be0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37239 } } peers { permanent_uuid: "c3fa4f56c1e1420c98b0979ec6066a9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35495 } }
I20260619 14:41:17.727880 18456 leader_election.cc:288] T c4303f5676d74e3d9cb57b3d4fbff86a P 912eb9fd34324a53a521635acb492be0 [CANDIDATE]: Term 1 election: Requested vote from peers e0e2af5a4ac440e8b40a9a0686eb5806 (127.0.98.1:36049), c3fa4f56c1e1420c98b0979ec6066a9f (127.0.98.2:35495)
I20260619 14:41:17.728209 18125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4303f5676d74e3d9cb57b3d4fbff86a" candidate_uuid: "912eb9fd34324a53a521635acb492be0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0e2af5a4ac440e8b40a9a0686eb5806"
I20260619 14:41:17.728325 18256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4303f5676d74e3d9cb57b3d4fbff86a" candidate_uuid: "912eb9fd34324a53a521635acb492be0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3fa4f56c1e1420c98b0979ec6066a9f"
I20260619 14:41:17.728334 18125 raft_consensus.cc:3037] T c4303f5676d74e3d9cb57b3d4fbff86a P e0e2af5a4ac440e8b40a9a0686eb5806 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:41:17.728442 18256 raft_consensus.cc:3037] T c4303f5676d74e3d9cb57b3d4fbff86a P c3fa4f56c1e1420c98b0979ec6066a9f [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:41:17.729653 18125 raft_consensus.cc:2454] T c4303f5676d74e3d9cb57b3d4fbff86a P e0e2af5a4ac440e8b40a9a0686eb5806 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 912eb9fd34324a53a521635acb492be0 in term 1.
I20260619 14:41:17.729653 18256 raft_consensus.cc:2454] T c4303f5676d74e3d9cb57b3d4fbff86a P c3fa4f56c1e1420c98b0979ec6066a9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 912eb9fd34324a53a521635acb492be0 in term 1.
I20260619 14:41:17.729954 18323 leader_election.cc:302] T c4303f5676d74e3d9cb57b3d4fbff86a P 912eb9fd34324a53a521635acb492be0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 912eb9fd34324a53a521635acb492be0, e0e2af5a4ac440e8b40a9a0686eb5806; no voters:
I20260619 14:41:17.730126 18456 raft_consensus.cc:2781] T c4303f5676d74e3d9cb57b3d4fbff86a P 912eb9fd34324a53a521635acb492be0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:41:17.730322 18456 raft_consensus.cc:686] T c4303f5676d74e3d9cb57b3d4fbff86a P 912eb9fd34324a53a521635acb492be0 [term 1 LEADER]: Becoming Leader. State: Replica: 912eb9fd34324a53a521635acb492be0, State: Running, Role: LEADER
I20260619 14:41:17.730633 18456 consensus_queue.cc:237] T c4303f5676d74e3d9cb57b3d4fbff86a P 912eb9fd34324a53a521635acb492be0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e2af5a4ac440e8b40a9a0686eb5806" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36049 } } peers { permanent_uuid: "912eb9fd34324a53a521635acb492be0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37239 } } peers { permanent_uuid: "c3fa4f56c1e1420c98b0979ec6066a9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35495 } }
I20260619 14:41:17.733922 17982 catalog_manager.cc:5495] T c4303f5676d74e3d9cb57b3d4fbff86a P 912eb9fd34324a53a521635acb492be0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 912eb9fd34324a53a521635acb492be0 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "912eb9fd34324a53a521635acb492be0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e2af5a4ac440e8b40a9a0686eb5806" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "912eb9fd34324a53a521635acb492be0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3fa4f56c1e1420c98b0979ec6066a9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35495 } health_report { overall_health: UNKNOWN } } }
I20260619 14:41:17.756947 18449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:17.753528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41364 (local address 127.0.98.3:37239)
0619 14:41:17.753655 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:17.753659 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:17.753674 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:41:17.753794 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:17.753797 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:17.753849 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:17.753950 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:41:17.754186 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:17.754694 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:17.755517 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:17.755734 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:17.755818 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:17.756107 (+ 289us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:41:17.756526 (+ 419us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:41:17.756635 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:17.756700 (+ 65us) server_negotiation.cc:299] Negotiation successful
0619 14:41:17.756780 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 14:41:17.762460 18256 raft_consensus.cc:1261] T c4303f5676d74e3d9cb57b3d4fbff86a P c3fa4f56c1e1420c98b0979ec6066a9f [term 1 FOLLOWER]: Refusing update from remote peer 912eb9fd34324a53a521635acb492be0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:41:17.762475 18125 raft_consensus.cc:1261] T c4303f5676d74e3d9cb57b3d4fbff86a P e0e2af5a4ac440e8b40a9a0686eb5806 [term 1 FOLLOWER]: Refusing update from remote peer 912eb9fd34324a53a521635acb492be0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260619 14:41:17.763037 18172 tablet.cc:2369] T c4303f5676d74e3d9cb57b3d4fbff86a P e0e2af5a4ac440e8b40a9a0686eb5806: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:41:17.763155 18456 consensus_queue.cc:1035] T c4303f5676d74e3d9cb57b3d4fbff86a P 912eb9fd34324a53a521635acb492be0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0e2af5a4ac440e8b40a9a0686eb5806" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:41:17.763357 18460 consensus_queue.cc:1035] T c4303f5676d74e3d9cb57b3d4fbff86a P 912eb9fd34324a53a521635acb492be0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3fa4f56c1e1420c98b0979ec6066a9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:41:17.771246 18465 mvcc.cc:205] Tried to move back new op lower bound from 7298580798508519424 to 7298580798387466240. Current Snapshot: MvccSnapshot[applied={T|T < 7298580798508519424}]
I20260619 14:41:17.781003 17981 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:41:17.781196 17981 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:41:17.783517 17981 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3fec07b64bd64536b4efac69fa80907b: Sending DeleteTablet for 3 replicas of tablet c4303f5676d74e3d9cb57b3d4fbff86a
I20260619 14:41:17.784296 18367 tablet_service.cc:1514] Processing DeleteTablet for tablet c4303f5676d74e3d9cb57b3d4fbff86a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:41:17 UTC) from {username='slave'} at 127.0.0.1:41350
I20260619 14:41:17.784291 18105 tablet_service.cc:1514] Processing DeleteTablet for tablet c4303f5676d74e3d9cb57b3d4fbff86a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:41:17 UTC) from {username='slave'} at 127.0.0.1:37054
I20260619 14:41:17.784358 18236 tablet_service.cc:1514] Processing DeleteTablet for tablet c4303f5676d74e3d9cb57b3d4fbff86a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:41:17 UTC) from {username='slave'} at 127.0.0.1:59798
I20260619 14:41:17.784849 18473 tablet_replica.cc:330] T c4303f5676d74e3d9cb57b3d4fbff86a P e0e2af5a4ac440e8b40a9a0686eb5806: stopping tablet replica
I20260619 14:41:17.785184 18473 raft_consensus.cc:2229] T c4303f5676d74e3d9cb57b3d4fbff86a P e0e2af5a4ac440e8b40a9a0686eb5806 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:41:17.785401 18475 tablet_replica.cc:330] T c4303f5676d74e3d9cb57b3d4fbff86a P c3fa4f56c1e1420c98b0979ec6066a9f: stopping tablet replica
I20260619 14:41:17.785507 18473 raft_consensus.cc:2258] T c4303f5676d74e3d9cb57b3d4fbff86a P e0e2af5a4ac440e8b40a9a0686eb5806 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:41:17.785553 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 18043
I20260619 14:41:17.786260 18475 raft_consensus.cc:2229] T c4303f5676d74e3d9cb57b3d4fbff86a P c3fa4f56c1e1420c98b0979ec6066a9f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:41:17.786548 18475 raft_consensus.cc:2258] T c4303f5676d74e3d9cb57b3d4fbff86a P c3fa4f56c1e1420c98b0979ec6066a9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:41:17.786991 18474 tablet_replica.cc:330] T c4303f5676d74e3d9cb57b3d4fbff86a P 912eb9fd34324a53a521635acb492be0: stopping tablet replica
I20260619 14:41:17.787245 18474 raft_consensus.cc:2229] T c4303f5676d74e3d9cb57b3d4fbff86a P 912eb9fd34324a53a521635acb492be0 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:41:17.787611 18474 raft_consensus.cc:2258] T c4303f5676d74e3d9cb57b3d4fbff86a P 912eb9fd34324a53a521635acb492be0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:41:17.789037 18474 ts_tablet_manager.cc:1905] T c4303f5676d74e3d9cb57b3d4fbff86a P 912eb9fd34324a53a521635acb492be0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:41:17.789314 18475 ts_tablet_manager.cc:1905] T c4303f5676d74e3d9cb57b3d4fbff86a P c3fa4f56c1e1420c98b0979ec6066a9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:41:17.791656 18474 ts_tablet_manager.cc:1918] T c4303f5676d74e3d9cb57b3d4fbff86a P 912eb9fd34324a53a521635acb492be0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:41:17.791750 18474 log.cc:1196] T c4303f5676d74e3d9cb57b3d4fbff86a P 912eb9fd34324a53a521635acb492be0: Deleting WAL directory at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/ts-2/wal/wals/c4303f5676d74e3d9cb57b3d4fbff86a
I20260619 14:41:17.791887 18475 ts_tablet_manager.cc:1918] T c4303f5676d74e3d9cb57b3d4fbff86a P c3fa4f56c1e1420c98b0979ec6066a9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:41:17.791980 18475 log.cc:1196] T c4303f5676d74e3d9cb57b3d4fbff86a P c3fa4f56c1e1420c98b0979ec6066a9f: Deleting WAL directory at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781879837648036-392-0/minicluster-data/ts-1/wal/wals/c4303f5676d74e3d9cb57b3d4fbff86a
I20260619 14:41:17.792045 18474 ts_tablet_manager.cc:1939] T c4303f5676d74e3d9cb57b3d4fbff86a P 912eb9fd34324a53a521635acb492be0: Deleting consensus metadata
I20260619 14:41:17.792248 18475 ts_tablet_manager.cc:1939] T c4303f5676d74e3d9cb57b3d4fbff86a P c3fa4f56c1e1420c98b0979ec6066a9f: Deleting consensus metadata
I20260619 14:41:17.793238 17967 catalog_manager.cc:4841] TS 912eb9fd34324a53a521635acb492be0 (127.0.98.3:37239): tablet c4303f5676d74e3d9cb57b3d4fbff86a (table test-table [id=ee6eed2c5d6a4cc1815a0c2b5c54ccaa]) successfully deleted
I20260619 14:41:17.793490 17969 catalog_manager.cc:4841] TS c3fa4f56c1e1420c98b0979ec6066a9f (127.0.98.2:35495): tablet c4303f5676d74e3d9cb57b3d4fbff86a (table test-table [id=ee6eed2c5d6a4cc1815a0c2b5c54ccaa]) successfully deleted
W20260619 14:41:17.794234 17970 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36049 (error 108)
I20260619 14:41:17.795028 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 18174
I20260619 14:41:17.795280 18446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:17.794982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36049 (local address 127.0.0.1:55132)
0619 14:41:17.795122 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:17.795197 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 14:41:17.795778 17970 catalog_manager.cc:4584] TS e0e2af5a4ac440e8b40a9a0686eb5806 (127.0.98.1:36049): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c4303f5676d74e3d9cb57b3d4fbff86a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36049: connect: Connection refused (error 111)
I20260619 14:41:17.800619 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 18305
I20260619 14:41:17.805725 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 17952
2026-06-19T14:41:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (991 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:41:17 dist-test-slave-32hj krb5kdc[18479](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:41:17 dist-test-slave-32hj krb5kdc[18479](info): set up 2 sockets
Jun 19 14:41:17 dist-test-slave-32hj krb5kdc[18479](info): commencing operation
krb5kdc: starting...
W20260619 14:41:18.760353 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/krb5kdc/test-user.keytab.
Jun 19 14:41:18 dist-test-slave-32hj krb5kdc[18479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:41:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:41:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:41:18.879114 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39523
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39523
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:41:18.985512 18492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:41:18.985785 18492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:41:18.985826 18492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:41:18.989778 18492 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:41:18.989840 18492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:41:18.989858 18492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:41:18.989874 18492 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:41:18.989887 18492 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:41:18.996308 18492 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:41:18.997605 18492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:41:18.998930 18492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:41:19.007953 18498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:19.007997 18497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:19.007977 18500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:19.008577 18492 server_base.cc:1034] running on GCE node
I20260619 14:41:19.009268 18492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:41:19.010453 18492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:41:19.011628 18492 hybrid_clock.cc:648] HybridClock initialized: now 1781880079011599 us; error 44 us; skew 500 ppm
Jun 19 14:41:19 dist-test-slave-32hj krb5kdc[18479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880079, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:41:19.014658 18492 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:41:19.015631 18492 webserver.cc:460] Webserver started at http://127.0.98.62:37543/ using document root <none> and password file <none>
I20260619 14:41:19.016218 18492 fs_manager.cc:359] Metadata directory not provided
I20260619 14:41:19.016291 18492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:41:19.016510 18492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:41:19.018716 18492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/master-0/data/instance:
uuid: "0a39a240789e48daa1ac66aafd70e7fd"
format_stamp: "Formatted at 2026-06-19 14:41:19 on dist-test-slave-32hj"
server_key: "2f00c0d3ae344b071e0ed7232c0b1724"
server_key_iv: "6a969975486acf9e4bef5f5a3dbb5acc"
server_key_version: "encryptionkey@0"
I20260619 14:41:19.019284 18492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/master-0/wal/instance:
uuid: "0a39a240789e48daa1ac66aafd70e7fd"
format_stamp: "Formatted at 2026-06-19 14:41:19 on dist-test-slave-32hj"
server_key: "2f00c0d3ae344b071e0ed7232c0b1724"
server_key_iv: "6a969975486acf9e4bef5f5a3dbb5acc"
server_key_version: "encryptionkey@0"
I20260619 14:41:19.023072 18492 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260619 14:41:19.025628 18507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:19.026881 18492 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:41:19.027006 18492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/master-0/wal
uuid: "0a39a240789e48daa1ac66aafd70e7fd"
format_stamp: "Formatted at 2026-06-19 14:41:19 on dist-test-slave-32hj"
server_key: "2f00c0d3ae344b071e0ed7232c0b1724"
server_key_iv: "6a969975486acf9e4bef5f5a3dbb5acc"
server_key_version: "encryptionkey@0"
I20260619 14:41:19.027110 18492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:41:19.063469 18492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:41:19.066413 18492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:41:19.066598 18492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:41:19.073501 18492 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39523
I20260619 14:41:19.073560 18559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39523 every 8 connection(s)
I20260619 14:41:19.074505 18492 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:41:19.075219 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 18492
I20260619 14:41:19.075330 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/master-0/wal/instance
I20260619 14:41:19.075585 392 external_mini_cluster.cc:1428] Setting key 052aeaf9841e612d3424fd0906213d0e
I20260619 14:41:19.077615 18560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:41:19.084071 18560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a39a240789e48daa1ac66aafd70e7fd: Bootstrap starting.
Jun 19 14:41:19 dist-test-slave-32hj krb5kdc[18479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:41:19.086380 18560 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a39a240789e48daa1ac66aafd70e7fd: Neither blocks nor log segments found. Creating new log.
I20260619 14:41:19.087121 18560 log.cc:824] T 00000000000000000000000000000000 P 0a39a240789e48daa1ac66aafd70e7fd: Log is configured to *not* fsync() on all Append() calls
I20260619 14:41:19.089126 18560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a39a240789e48daa1ac66aafd70e7fd: No bootstrap required, opened a new log
I20260619 14:41:19.089833 18563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:19.077070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51524 (local address 127.0.98.62:39523)
0619 14:41:19.077467 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:19.077484 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:19.077520 (+ 36us) server_negotiation.cc:407] Connection header received
0619 14:41:19.078188 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:19.078205 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:19.078548 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:19.078842 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:19.079729 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:19.080813 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:19.081952 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:19.082413 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:19.082523 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:19.085274 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:19.085279 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:19.085293 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:41:19.085325 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:19.086968 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:19.087421 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:19.087426 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:19.087431 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:19.087548 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:19.087809 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:19.087813 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:19.087815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:19.088253 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:19.088437 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:19.088760 (+ 323us) server_negotiation.cc:299] Negotiation successful
0619 14:41:19.089050 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":111,"threads_started":1}
I20260619 14:41:19.093044 18560 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a39a240789e48daa1ac66aafd70e7fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a39a240789e48daa1ac66aafd70e7fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39523 } }
I20260619 14:41:19.093281 18560 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a39a240789e48daa1ac66aafd70e7fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:41:19.093338 18560 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a39a240789e48daa1ac66aafd70e7fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a39a240789e48daa1ac66aafd70e7fd, State: Initialized, Role: FOLLOWER
I20260619 14:41:19.093734 18560 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a39a240789e48daa1ac66aafd70e7fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a39a240789e48daa1ac66aafd70e7fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39523 } }
I20260619 14:41:19.093871 18560 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a39a240789e48daa1ac66aafd70e7fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:41:19.093932 18560 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a39a240789e48daa1ac66aafd70e7fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:41:19.094030 18560 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a39a240789e48daa1ac66aafd70e7fd [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:41:19.094959 18560 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a39a240789e48daa1ac66aafd70e7fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a39a240789e48daa1ac66aafd70e7fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39523 } }
I20260619 14:41:19.095288 18560 leader_election.cc:302] T 00000000000000000000000000000000 P 0a39a240789e48daa1ac66aafd70e7fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a39a240789e48daa1ac66aafd70e7fd; no voters:
I20260619 14:41:19.095597 18560 leader_election.cc:288] T 00000000000000000000000000000000 P 0a39a240789e48daa1ac66aafd70e7fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:41:19.095717 18565 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a39a240789e48daa1ac66aafd70e7fd [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:41:19.095944 18565 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a39a240789e48daa1ac66aafd70e7fd [term 1 LEADER]: Becoming Leader. State: Replica: 0a39a240789e48daa1ac66aafd70e7fd, State: Running, Role: LEADER
I20260619 14:41:19.096261 18565 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a39a240789e48daa1ac66aafd70e7fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a39a240789e48daa1ac66aafd70e7fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39523 } }
I20260619 14:41:19.096863 18560 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a39a240789e48daa1ac66aafd70e7fd [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:41:19.097892 18566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a39a240789e48daa1ac66aafd70e7fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a39a240789e48daa1ac66aafd70e7fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a39a240789e48daa1ac66aafd70e7fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39523 } } }
I20260619 14:41:19.098057 18566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a39a240789e48daa1ac66aafd70e7fd [sys.catalog]: This master's current role is: LEADER
I20260619 14:41:19.097891 18567 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a39a240789e48daa1ac66aafd70e7fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a39a240789e48daa1ac66aafd70e7fd. Latest consensus state: current_term: 1 leader_uuid: "0a39a240789e48daa1ac66aafd70e7fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a39a240789e48daa1ac66aafd70e7fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39523 } } }
I20260619 14:41:19.098233 18567 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a39a240789e48daa1ac66aafd70e7fd [sys.catalog]: This master's current role is: LEADER
I20260619 14:41:19.098455 18574 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:41:19.101600 18574 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:41:19.107223 18574 catalog_manager.cc:1347] Generated new cluster ID: e323fb54287141bf8ebb32824c0787af
I20260619 14:41:19.107306 18574 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:41:19.119140 18574 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:41:19.120445 18574 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:41:19.130697 18574 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a39a240789e48daa1ac66aafd70e7fd: Generated new TSK 0
I20260619 14:41:19.131605 18574 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 14:41:19.169366 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39523
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:41:19.276903 18586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:41:19.277165 18586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:41:19.277215 18586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:41:19.281183 18586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:41:19.281255 18586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:41:19.281314 18586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:41:19.286613 18586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:41:19.287638 18586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:41:19.288616 18586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:41:19.298240 18586 server_base.cc:1034] running on GCE node
W20260619 14:41:19.298277 18591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:19.298173 18592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:19.298255 18594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:19.298803 18586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:41:19.299470 18586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:41:19.300642 18586 hybrid_clock.cc:648] HybridClock initialized: now 1781880079300606 us; error 49 us; skew 500 ppm
Jun 19 14:41:19 dist-test-slave-32hj krb5kdc[18479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880079, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:41:19.303336 18586 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:41:19.304179 18586 webserver.cc:460] Webserver started at http://127.0.98.1:35395/ using document root <none> and password file <none>
I20260619 14:41:19.304792 18586 fs_manager.cc:359] Metadata directory not provided
I20260619 14:41:19.304862 18586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:41:19.305053 18586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:41:19.307027 18586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/ts-0/data/instance:
uuid: "7b0b3627e30d4a15b1fe9ff2699d2e43"
format_stamp: "Formatted at 2026-06-19 14:41:19 on dist-test-slave-32hj"
server_key: "b1c107fb4a750b865537d09cedefeb99"
server_key_iv: "c317ff22ac6191cb2fb2d4a275917d56"
server_key_version: "encryptionkey@0"
I20260619 14:41:19.307547 18586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7b0b3627e30d4a15b1fe9ff2699d2e43"
format_stamp: "Formatted at 2026-06-19 14:41:19 on dist-test-slave-32hj"
server_key: "b1c107fb4a750b865537d09cedefeb99"
server_key_iv: "c317ff22ac6191cb2fb2d4a275917d56"
server_key_version: "encryptionkey@0"
I20260619 14:41:19.311081 18586 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:41:19.313364 18601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:19.314486 18586 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:41:19.314587 18586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/ts-0/wal
uuid: "7b0b3627e30d4a15b1fe9ff2699d2e43"
format_stamp: "Formatted at 2026-06-19 14:41:19 on dist-test-slave-32hj"
server_key: "b1c107fb4a750b865537d09cedefeb99"
server_key_iv: "c317ff22ac6191cb2fb2d4a275917d56"
server_key_version: "encryptionkey@0"
I20260619 14:41:19.314672 18586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:41:19.328570 18586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:41:19.331509 18586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:41:19.331681 18586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:41:19.332242 18586 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:41:19.333084 18586 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:41:19.333130 18586 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:19.333169 18586 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:41:19.333184 18586 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:19.341662 18586 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43187
I20260619 14:41:19.341704 18714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43187 every 8 connection(s)
I20260619 14:41:19.342525 18586 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:41:19.345372 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 18586
I20260619 14:41:19.345498 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:41:19.345826 392 external_mini_cluster.cc:1428] Setting key 9beb2dd1605f21ac7f1dfab6c7c5c1b3
Jun 19 14:41:19 dist-test-slave-32hj krb5kdc[18479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880079, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 14:41:19.356009 18563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:19.344205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50335 (local address 127.0.98.62:39523)
0619 14:41:19.344343 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:19.344347 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:19.345296 (+ 949us) server_negotiation.cc:407] Connection header received
0619 14:41:19.346758 (+ 1462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:19.346761 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:19.346804 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:19.346887 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:19.348313 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:19.348980 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:19.350078 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:19.350299 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:19.350394 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:19.352662 (+ 2268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:19.352665 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:19.352667 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:41:19.352690 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:19.354071 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:19.354586 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:19.354589 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:19.354590 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:19.354659 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:19.354967 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:19.354970 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:19.354971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:19.355139 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:19.355223 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:19.355713 (+ 490us) server_negotiation.cc:299] Negotiation successful
0619 14:41:19.355844 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 14:41:19.356936 18717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:19.344641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39523 (local address 127.0.98.1:50335)
0619 14:41:19.345126 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:19.345183 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:19.346572 (+ 1389us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:19.347048 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:19.347057 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:19.347513 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:41:19.348175 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:19.348193 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:19.349091 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:19.349093 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:19.349961 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:19.349970 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:19.350430 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:19.350435 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:19.350589 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:19.351192 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:41:19.351208 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:41:19.352515 (+ 1307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:41:19.354176 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:41:19.354181 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:41:19.354195 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:41:19.354493 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:41:19.354739 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:41:19.354742 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:41:19.354744 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:41:19.354881 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:41:19.355256 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:41:19.355263 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:41:19.355495 (+ 232us) client_negotiation.cc:769] Sending connection context
0619 14:41:19.355655 (+ 160us) client_negotiation.cc:240] Negotiation successful
0619 14:41:19.356006 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":110,"threads_started":1}
I20260619 14:41:19.358197 18715 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39523
I20260619 14:41:19.358467 18715 heartbeater.cc:464] Registering TS with master...
I20260619 14:41:19.359035 18715 heartbeater.cc:511] Master 127.0.98.62:39523 requested a full tablet report, sending...
I20260619 14:41:19.360637 18524 ts_manager.cc:194] Registered new tserver with Master: 7b0b3627e30d4a15b1fe9ff2699d2e43 (127.0.98.1:43187)
I20260619 14:41:19.361759 18524 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50335
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 14:41:19.371657 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39523
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:41:19.477245 18720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:41:19.477499 18720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:41:19.477557 18720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:41:19.481459 18720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:41:19.481513 18720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:41:19.481588 18720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:41:19.486630 18720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:41:19.487630 18720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:41:19.488592 18720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:41:19.497519 18726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:19.497504 18725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:19.497504 18728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:19.497645 18720 server_base.cc:1034] running on GCE node
I20260619 14:41:19.498119 18720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:41:19.498710 18720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:41:19.499864 18720 hybrid_clock.cc:648] HybridClock initialized: now 1781880079499850 us; error 29 us; skew 500 ppm
Jun 19 14:41:19 dist-test-slave-32hj krb5kdc[18479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880079, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:41:19.502279 18720 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:41:19.503028 18720 webserver.cc:460] Webserver started at http://127.0.98.2:37165/ using document root <none> and password file <none>
I20260619 14:41:19.503518 18720 fs_manager.cc:359] Metadata directory not provided
I20260619 14:41:19.503580 18720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:41:19.503750 18720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:41:19.505764 18720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/ts-1/data/instance:
uuid: "82a5d299d90a4840bd60371f7c9e3747"
format_stamp: "Formatted at 2026-06-19 14:41:19 on dist-test-slave-32hj"
server_key: "d79cc2a845af9eefe9e6e788d40d0467"
server_key_iv: "d6dedae0c359849a04a99ae5e04ca2de"
server_key_version: "encryptionkey@0"
I20260619 14:41:19.506276 18720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/ts-1/wal/instance:
uuid: "82a5d299d90a4840bd60371f7c9e3747"
format_stamp: "Formatted at 2026-06-19 14:41:19 on dist-test-slave-32hj"
server_key: "d79cc2a845af9eefe9e6e788d40d0467"
server_key_iv: "d6dedae0c359849a04a99ae5e04ca2de"
server_key_version: "encryptionkey@0"
I20260619 14:41:19.509646 18720 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:41:19.511783 18735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:19.512928 18720 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260619 14:41:19.513033 18720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/ts-1/wal
uuid: "82a5d299d90a4840bd60371f7c9e3747"
format_stamp: "Formatted at 2026-06-19 14:41:19 on dist-test-slave-32hj"
server_key: "d79cc2a845af9eefe9e6e788d40d0467"
server_key_iv: "d6dedae0c359849a04a99ae5e04ca2de"
server_key_version: "encryptionkey@0"
I20260619 14:41:19.513118 18720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:41:19.526325 18720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:41:19.529038 18720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:41:19.529199 18720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:41:19.529734 18720 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:41:19.530534 18720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:41:19.530579 18720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:19.530628 18720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:41:19.530655 18720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:19.538765 18720 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45247
I20260619 14:41:19.538815 18848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45247 every 8 connection(s)
I20260619 14:41:19.539647 18720 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:41:19.547366 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 18720
I20260619 14:41:19.547500 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:41:19.547819 392 external_mini_cluster.cc:1428] Setting key fdb6e8826f85b4c5c3cccda2fe272e4d
Jun 19 14:41:19 dist-test-slave-32hj krb5kdc[18479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880079, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:41:19.552498 18563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:19.541501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60029 (local address 127.0.98.62:39523)
0619 14:41:19.541652 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:19.541655 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:19.542442 (+ 787us) server_negotiation.cc:407] Connection header received
0619 14:41:19.543360 (+ 918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:19.543363 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:19.543419 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:19.543493 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:19.544633 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:19.545205 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:19.546312 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:19.546541 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:19.546627 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:19.549266 (+ 2639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:19.549270 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:19.549273 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:41:19.549298 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:19.550645 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:19.551116 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:19.551119 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:19.551120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:19.551195 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:19.551502 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:19.551505 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:19.551507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:19.551677 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:19.551761 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:19.552228 (+ 467us) server_negotiation.cc:299] Negotiation successful
0619 14:41:19.552345 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 14:41:19.553351 18851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:19.541832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39523 (local address 127.0.98.2:60029)
0619 14:41:19.542289 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:19.542333 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:19.543150 (+ 817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:19.543670 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:19.543677 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:19.544028 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:41:19.544496 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:19.544510 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:19.545319 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:19.545322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:19.546199 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:19.546208 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:19.546676 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:19.546680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:19.546835 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:19.547398 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:41:19.547418 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:41:19.549124 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:41:19.550754 (+ 1630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:41:19.550759 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:41:19.550770 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:41:19.551014 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:41:19.551276 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:41:19.551279 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:41:19.551281 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:41:19.551415 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:41:19.551770 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:41:19.551776 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:41:19.552005 (+ 229us) client_negotiation.cc:769] Sending connection context
0619 14:41:19.552167 (+ 162us) client_negotiation.cc:240] Negotiation successful
0619 14:41:19.552485 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":109,"threads_started":1}
I20260619 14:41:19.554432 18849 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39523
I20260619 14:41:19.554701 18849 heartbeater.cc:464] Registering TS with master...
I20260619 14:41:19.555255 18849 heartbeater.cc:511] Master 127.0.98.62:39523 requested a full tablet report, sending...
I20260619 14:41:19.556339 18524 ts_manager.cc:194] Registered new tserver with Master: 82a5d299d90a4840bd60371f7c9e3747 (127.0.98.2:45247)
I20260619 14:41:19.556910 18524 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60029
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 14:41:19.575255 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39523
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:41:19.681037 18854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:41:19.681282 18854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:41:19.681329 18854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:41:19.685295 18854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:41:19.685361 18854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:41:19.685426 18854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:41:19.690681 18854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:41:19.691685 18854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:41:19.692663 18854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:41:19.701895 18859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:19.701931 18854 server_base.cc:1034] running on GCE node
W20260619 14:41:19.702023 18860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:19.701930 18862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:19.702523 18854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:41:19.703112 18854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:41:19.704259 18854 hybrid_clock.cc:648] HybridClock initialized: now 1781880079704240 us; error 30 us; skew 500 ppm
Jun 19 14:41:19 dist-test-slave-32hj krb5kdc[18479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880079, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:41:19.706915 18854 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:41:19.707762 18854 webserver.cc:460] Webserver started at http://127.0.98.3:37573/ using document root <none> and password file <none>
I20260619 14:41:19.708282 18854 fs_manager.cc:359] Metadata directory not provided
I20260619 14:41:19.708367 18854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:41:19.708567 18854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:41:19.710580 18854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/ts-2/data/instance:
uuid: "0c725233c7f24cd78388a0209f5ab49c"
format_stamp: "Formatted at 2026-06-19 14:41:19 on dist-test-slave-32hj"
server_key: "0382d60632cab8a11b5f1bd02348d2a4"
server_key_iv: "963897f75054688a52db6e12d54cb667"
server_key_version: "encryptionkey@0"
I20260619 14:41:19.711095 18854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0c725233c7f24cd78388a0209f5ab49c"
format_stamp: "Formatted at 2026-06-19 14:41:19 on dist-test-slave-32hj"
server_key: "0382d60632cab8a11b5f1bd02348d2a4"
server_key_iv: "963897f75054688a52db6e12d54cb667"
server_key_version: "encryptionkey@0"
I20260619 14:41:19.714617 18854 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260619 14:41:19.716943 18869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:19.718165 18854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:41:19.718266 18854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/ts-2/wal
uuid: "0c725233c7f24cd78388a0209f5ab49c"
format_stamp: "Formatted at 2026-06-19 14:41:19 on dist-test-slave-32hj"
server_key: "0382d60632cab8a11b5f1bd02348d2a4"
server_key_iv: "963897f75054688a52db6e12d54cb667"
server_key_version: "encryptionkey@0"
I20260619 14:41:19.718385 18854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:41:19.738763 18854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:41:19.741660 18854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:41:19.741834 18854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:41:19.742440 18854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:41:19.743250 18854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:41:19.743295 18854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:19.743337 18854 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:41:19.743376 18854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:19.751952 18854 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45777
I20260619 14:41:19.752022 18982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45777 every 8 connection(s)
I20260619 14:41:19.752885 18854 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:41:19.761204 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 18854
I20260619 14:41:19.761314 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/ts-2/wal/instance
Jun 19 14:41:19 dist-test-slave-32hj krb5kdc[18479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880079, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:41:19.761611 392 external_mini_cluster.cc:1428] Setting key 29a8fc2c18e0928b317531fa0962f88e
I20260619 14:41:19.765515 18563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:19.754632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59089 (local address 127.0.98.62:39523)
0619 14:41:19.754773 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:19.754778 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:19.755662 (+ 884us) server_negotiation.cc:407] Connection header received
0619 14:41:19.756492 (+ 830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:19.756495 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:19.756543 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:19.756614 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:19.757765 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:19.758271 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:19.759400 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:19.759611 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:19.759688 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:19.762211 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:19.762214 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:19.762216 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:41:19.762239 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:19.763727 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:19.764177 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:19.764180 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:19.764182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:19.764258 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:19.764563 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:19.764565 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:19.764566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:19.764779 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:19.764866 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:19.765300 (+ 434us) server_negotiation.cc:299] Negotiation successful
0619 14:41:19.765398 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 14:41:19.766279 18985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:19.755054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39523 (local address 127.0.98.3:59089)
0619 14:41:19.755506 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:19.755549 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:19.756304 (+ 755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:19.756810 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:19.756817 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:19.757185 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:41:19.757648 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:19.757663 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:19.758372 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:19.758375 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:19.759285 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:19.759294 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:19.759740 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:19.759743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:19.759878 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:19.760435 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:41:19.760450 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:41:19.762053 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:41:19.763827 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:41:19.763831 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:41:19.763843 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:41:19.764077 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:41:19.764360 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:41:19.764362 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:41:19.764363 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:41:19.764495 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:41:19.764878 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:41:19.764883 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:41:19.765110 (+ 227us) client_negotiation.cc:769] Sending connection context
0619 14:41:19.765262 (+ 152us) client_negotiation.cc:240] Negotiation successful
0619 14:41:19.765522 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":109,"threads_started":1}
I20260619 14:41:19.767252 18983 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39523
I20260619 14:41:19.767468 18983 heartbeater.cc:464] Registering TS with master...
I20260619 14:41:19.767921 18983 heartbeater.cc:511] Master 127.0.98.62:39523 requested a full tablet report, sending...
I20260619 14:41:19.768880 18524 ts_manager.cc:194] Registered new tserver with Master: 0c725233c7f24cd78388a0209f5ab49c (127.0.98.3:45777)
I20260619 14:41:19.769411 18524 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59089
I20260619 14:41:19.775079 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:41:19.783290 18563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:19.776859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51530 (local address 127.0.98.62:39523)
0619 14:41:19.776993 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:19.776997 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:19.777117 (+ 120us) server_negotiation.cc:407] Connection header received
0619 14:41:19.777206 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:19.777208 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:19.777242 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:19.777304 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:19.777911 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:19.778421 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:19.779204 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:19.779415 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:19.779491 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:19.780872 (+ 1381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:19.780876 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:19.780877 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:41:19.780898 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:19.781965 (+ 1067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:19.782366 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:19.782368 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:19.782369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:19.782433 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:19.782657 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:19.782660 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:19.782662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:19.782834 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:19.782937 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:19.783090 (+ 153us) server_negotiation.cc:299] Negotiation successful
0619 14:41:19.783179 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 14:41:19.785983 18524 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51530:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:41:19.788898 18524 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:41:19.802418 18995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:19.797061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43187 (local address 127.0.0.1:47684)
0619 14:41:19.797650 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:19.797674 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:19.797798 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:19.798543 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:19.798548 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:19.798569 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:41:19.798712 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:19.798720 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:19.799826 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:19.799829 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:19.800835 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:19.800863 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:19.801915 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:19.801918 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:19.802016 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:19.802049 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 14:41:19.802151 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 14:41:19.802268 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":118,"threads_started":1}
I20260619 14:41:19.802424 18998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:19.797783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47684 (local address 127.0.98.1:43187)
0619 14:41:19.798073 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:19.798082 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:19.798101 (+ 19us) server_negotiation.cc:407] Connection header received
0619 14:41:19.798161 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:19.798164 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:19.798378 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:19.798862 (+ 484us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:41:19.798994 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:19.799710 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:19.800986 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:19.801817 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:19.801909 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:19.802060 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:19.802196 (+ 136us) server_negotiation.cc:299] Negotiation successful
0619 14:41:19.802283 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":88,"threads_started":1}
I20260619 14:41:19.803069 18994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:19.796957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45247 (local address 127.0.0.1:52864)
0619 14:41:19.797622 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:19.797672 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:19.797812 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:19.798797 (+ 985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:19.798800 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:19.798817 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:41:19.798928 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:19.798934 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:19.800505 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:19.800508 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:19.801727 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:19.801737 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:19.802695 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:19.802698 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:19.802781 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:19.802803 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 14:41:19.802881 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 14:41:19.802938 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"mutex_wait_us":40,"spinlock_wait_cycles":1408,"thread_start_us":75,"threads_started":1}
I20260619 14:41:19.803159 18996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:19.797095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52864 (local address 127.0.98.2:45247)
0619 14:41:19.798373 (+ 1278us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:19.798383 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:19.798406 (+ 23us) server_negotiation.cc:407] Connection header received
0619 14:41:19.798466 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:19.798471 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:19.798647 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:19.799070 (+ 423us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:41:19.799231 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:19.800365 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:19.801848 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:19.802585 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:19.802682 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:19.802815 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:19.802913 (+ 98us) server_negotiation.cc:299] Negotiation successful
0619 14:41:19.803006 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1186,"thread_start_us":148,"threads_started":1}
I20260619 14:41:19.805671 18649 tablet_service.cc:1467] Processing CreateTablet for tablet 7e4e82aff6524ea8be3835e41516024d (DEFAULT_TABLE table=test-table [id=bf092b74f4fe44fbbae1f5432c3396af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:41:19.805667 18783 tablet_service.cc:1467] Processing CreateTablet for tablet 7e4e82aff6524ea8be3835e41516024d (DEFAULT_TABLE table=test-table [id=bf092b74f4fe44fbbae1f5432c3396af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:41:19.806573 18649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e4e82aff6524ea8be3835e41516024d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:41:19.806574 18783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e4e82aff6524ea8be3835e41516024d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:41:19.808391 18997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:19.797321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45777 (local address 127.0.0.1:42444)
0619 14:41:19.798027 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:19.798039 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:19.798114 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:19.798797 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:19.798800 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:19.798867 (+ 67us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:41:19.798931 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:19.798936 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:19.800530 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:19.800533 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:19.801648 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:19.801656 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:19.802890 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:19.802894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:19.802976 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:19.802997 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 14:41:19.803063 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 14:41:19.803124 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"mutex_wait_us":42,"spinlock_wait_cycles":2176,"thread_start_us":87,"threads_started":1}
I20260619 14:41:19.808949 18999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:19.797793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42444 (local address 127.0.98.3:45777)
0619 14:41:19.798330 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:19.798367 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:19.798391 (+ 24us) server_negotiation.cc:407] Connection header received
0619 14:41:19.798466 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:19.798471 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:19.798647 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:19.799068 (+ 421us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:41:19.799219 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:19.800368 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:19.801762 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:19.802575 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:19.802682 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:19.802815 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:19.808647 (+ 5832us) server_negotiation.cc:299] Negotiation successful
0619 14:41:19.808793 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":122,"threads_started":1}
I20260619 14:41:19.811894 18917 tablet_service.cc:1467] Processing CreateTablet for tablet 7e4e82aff6524ea8be3835e41516024d (DEFAULT_TABLE table=test-table [id=bf092b74f4fe44fbbae1f5432c3396af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:41:19.812547 19000 tablet_bootstrap.cc:492] T 7e4e82aff6524ea8be3835e41516024d P 7b0b3627e30d4a15b1fe9ff2699d2e43: Bootstrap starting.
I20260619 14:41:19.812997 18917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e4e82aff6524ea8be3835e41516024d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:41:19.814023 19001 tablet_bootstrap.cc:492] T 7e4e82aff6524ea8be3835e41516024d P 82a5d299d90a4840bd60371f7c9e3747: Bootstrap starting.
I20260619 14:41:19.814785 19000 tablet_bootstrap.cc:654] T 7e4e82aff6524ea8be3835e41516024d P 7b0b3627e30d4a15b1fe9ff2699d2e43: Neither blocks nor log segments found. Creating new log.
I20260619 14:41:19.815775 19000 log.cc:824] T 7e4e82aff6524ea8be3835e41516024d P 7b0b3627e30d4a15b1fe9ff2699d2e43: Log is configured to *not* fsync() on all Append() calls
I20260619 14:41:19.816085 19001 tablet_bootstrap.cc:654] T 7e4e82aff6524ea8be3835e41516024d P 82a5d299d90a4840bd60371f7c9e3747: Neither blocks nor log segments found. Creating new log.
I20260619 14:41:19.817320 19001 log.cc:824] T 7e4e82aff6524ea8be3835e41516024d P 82a5d299d90a4840bd60371f7c9e3747: Log is configured to *not* fsync() on all Append() calls
I20260619 14:41:19.817690 19000 tablet_bootstrap.cc:492] T 7e4e82aff6524ea8be3835e41516024d P 7b0b3627e30d4a15b1fe9ff2699d2e43: No bootstrap required, opened a new log
I20260619 14:41:19.817885 19000 ts_tablet_manager.cc:1397] T 7e4e82aff6524ea8be3835e41516024d P 7b0b3627e30d4a15b1fe9ff2699d2e43: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 14:41:19.819435 19001 tablet_bootstrap.cc:492] T 7e4e82aff6524ea8be3835e41516024d P 82a5d299d90a4840bd60371f7c9e3747: No bootstrap required, opened a new log
I20260619 14:41:19.819684 19001 ts_tablet_manager.cc:1397] T 7e4e82aff6524ea8be3835e41516024d P 82a5d299d90a4840bd60371f7c9e3747: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 14:41:19.820927 19004 tablet_bootstrap.cc:492] T 7e4e82aff6524ea8be3835e41516024d P 0c725233c7f24cd78388a0209f5ab49c: Bootstrap starting.
I20260619 14:41:19.821226 19000 raft_consensus.cc:348] T 7e4e82aff6524ea8be3835e41516024d P 7b0b3627e30d4a15b1fe9ff2699d2e43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a5d299d90a4840bd60371f7c9e3747" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45247 } } peers { permanent_uuid: "7b0b3627e30d4a15b1fe9ff2699d2e43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43187 } } peers { permanent_uuid: "0c725233c7f24cd78388a0209f5ab49c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45777 } }
I20260619 14:41:19.821445 19000 raft_consensus.cc:374] T 7e4e82aff6524ea8be3835e41516024d P 7b0b3627e30d4a15b1fe9ff2699d2e43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:41:19.821494 19000 raft_consensus.cc:729] T 7e4e82aff6524ea8be3835e41516024d P 7b0b3627e30d4a15b1fe9ff2699d2e43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b0b3627e30d4a15b1fe9ff2699d2e43, State: Initialized, Role: FOLLOWER
I20260619 14:41:19.821956 19000 consensus_queue.cc:260] T 7e4e82aff6524ea8be3835e41516024d P 7b0b3627e30d4a15b1fe9ff2699d2e43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a5d299d90a4840bd60371f7c9e3747" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45247 } } peers { permanent_uuid: "7b0b3627e30d4a15b1fe9ff2699d2e43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43187 } } peers { permanent_uuid: "0c725233c7f24cd78388a0209f5ab49c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45777 } }
I20260619 14:41:19.822597 18715 heartbeater.cc:503] Master 127.0.98.62:39523 was elected leader, sending a full tablet report...
I20260619 14:41:19.822896 19000 ts_tablet_manager.cc:1428] T 7e4e82aff6524ea8be3835e41516024d P 7b0b3627e30d4a15b1fe9ff2699d2e43: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 14:41:19.823135 19004 tablet_bootstrap.cc:654] T 7e4e82aff6524ea8be3835e41516024d P 0c725233c7f24cd78388a0209f5ab49c: Neither blocks nor log segments found. Creating new log.
I20260619 14:41:19.824177 19004 log.cc:824] T 7e4e82aff6524ea8be3835e41516024d P 0c725233c7f24cd78388a0209f5ab49c: Log is configured to *not* fsync() on all Append() calls
I20260619 14:41:19.824072 19001 raft_consensus.cc:348] T 7e4e82aff6524ea8be3835e41516024d P 82a5d299d90a4840bd60371f7c9e3747 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a5d299d90a4840bd60371f7c9e3747" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45247 } } peers { permanent_uuid: "7b0b3627e30d4a15b1fe9ff2699d2e43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43187 } } peers { permanent_uuid: "0c725233c7f24cd78388a0209f5ab49c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45777 } }
I20260619 14:41:19.824349 19001 raft_consensus.cc:374] T 7e4e82aff6524ea8be3835e41516024d P 82a5d299d90a4840bd60371f7c9e3747 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:41:19.824411 19001 raft_consensus.cc:729] T 7e4e82aff6524ea8be3835e41516024d P 82a5d299d90a4840bd60371f7c9e3747 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82a5d299d90a4840bd60371f7c9e3747, State: Initialized, Role: FOLLOWER
I20260619 14:41:19.824854 19001 consensus_queue.cc:260] T 7e4e82aff6524ea8be3835e41516024d P 82a5d299d90a4840bd60371f7c9e3747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a5d299d90a4840bd60371f7c9e3747" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45247 } } peers { permanent_uuid: "7b0b3627e30d4a15b1fe9ff2699d2e43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43187 } } peers { permanent_uuid: "0c725233c7f24cd78388a0209f5ab49c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45777 } }
I20260619 14:41:19.825511 18849 heartbeater.cc:503] Master 127.0.98.62:39523 was elected leader, sending a full tablet report...
I20260619 14:41:19.826068 19004 tablet_bootstrap.cc:492] T 7e4e82aff6524ea8be3835e41516024d P 0c725233c7f24cd78388a0209f5ab49c: No bootstrap required, opened a new log
I20260619 14:41:19.826260 19004 ts_tablet_manager.cc:1397] T 7e4e82aff6524ea8be3835e41516024d P 0c725233c7f24cd78388a0209f5ab49c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:41:19.826730 19001 ts_tablet_manager.cc:1428] T 7e4e82aff6524ea8be3835e41516024d P 82a5d299d90a4840bd60371f7c9e3747: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260619 14:41:19.830144 19004 raft_consensus.cc:348] T 7e4e82aff6524ea8be3835e41516024d P 0c725233c7f24cd78388a0209f5ab49c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a5d299d90a4840bd60371f7c9e3747" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45247 } } peers { permanent_uuid: "7b0b3627e30d4a15b1fe9ff2699d2e43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43187 } } peers { permanent_uuid: "0c725233c7f24cd78388a0209f5ab49c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45777 } }
I20260619 14:41:19.830421 19004 raft_consensus.cc:374] T 7e4e82aff6524ea8be3835e41516024d P 0c725233c7f24cd78388a0209f5ab49c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:41:19.830475 19004 raft_consensus.cc:729] T 7e4e82aff6524ea8be3835e41516024d P 0c725233c7f24cd78388a0209f5ab49c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c725233c7f24cd78388a0209f5ab49c, State: Initialized, Role: FOLLOWER
I20260619 14:41:19.830842 19004 consensus_queue.cc:260] T 7e4e82aff6524ea8be3835e41516024d P 0c725233c7f24cd78388a0209f5ab49c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a5d299d90a4840bd60371f7c9e3747" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45247 } } peers { permanent_uuid: "7b0b3627e30d4a15b1fe9ff2699d2e43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43187 } } peers { permanent_uuid: "0c725233c7f24cd78388a0209f5ab49c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45777 } }
I20260619 14:41:19.831480 18983 heartbeater.cc:503] Master 127.0.98.62:39523 was elected leader, sending a full tablet report...
I20260619 14:41:19.831712 19004 ts_tablet_manager.cc:1428] T 7e4e82aff6524ea8be3835e41516024d P 0c725233c7f24cd78388a0209f5ab49c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260619 14:41:19.843575 18716 tablet.cc:2369] T 7e4e82aff6524ea8be3835e41516024d P 7b0b3627e30d4a15b1fe9ff2699d2e43: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:41:19.881160 19005 raft_consensus.cc:482] T 7e4e82aff6524ea8be3835e41516024d P 7b0b3627e30d4a15b1fe9ff2699d2e43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:41:19.881389 19005 raft_consensus.cc:504] T 7e4e82aff6524ea8be3835e41516024d P 7b0b3627e30d4a15b1fe9ff2699d2e43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a5d299d90a4840bd60371f7c9e3747" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45247 } } peers { permanent_uuid: "7b0b3627e30d4a15b1fe9ff2699d2e43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43187 } } peers { permanent_uuid: "0c725233c7f24cd78388a0209f5ab49c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45777 } }
I20260619 14:41:19.882524 19005 leader_election.cc:288] T 7e4e82aff6524ea8be3835e41516024d P 7b0b3627e30d4a15b1fe9ff2699d2e43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82a5d299d90a4840bd60371f7c9e3747 (127.0.98.2:45247), 0c725233c7f24cd78388a0209f5ab49c (127.0.98.3:45777)
I20260619 14:41:19.887651 18996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:19.882830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52273 (local address 127.0.98.2:45247)
0619 14:41:19.882961 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:19.882966 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:19.883170 (+ 204us) server_negotiation.cc:407] Connection header received
0619 14:41:19.883323 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:19.883326 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:19.883372 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:19.883446 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:41:19.883777 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:19.884573 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:19.886408 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:19.887039 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:19.887151 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:19.887218 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:19.887444 (+ 226us) server_negotiation.cc:299] Negotiation successful
0619 14:41:19.887517 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 14:41:19.887651 18999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:19.882890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54905 (local address 127.0.98.3:45777)
0619 14:41:19.883013 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:19.883018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:19.883200 (+ 182us) server_negotiation.cc:407] Connection header received
0619 14:41:19.883318 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:19.883321 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:19.883372 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:19.883446 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:41:19.883778 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:19.884548 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:19.886408 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:19.887053 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:19.887151 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:19.887218 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:19.887442 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 14:41:19.887517 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 14:41:19.887705 19010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:19.882789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45777 (local address 127.0.98.1:54905)
0619 14:41:19.883122 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:19.883133 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:19.883214 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:19.883465 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:19.883467 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:19.883503 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:41:19.883668 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:19.883674 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:19.884659 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:19.884663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:19.886260 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:19.886273 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:19.887158 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:19.887162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:19.887317 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:19.887356 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 14:41:19.887416 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 14:41:19.887521 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":267,"spinlock_wait_cycles":10496,"thread_start_us":125,"threads_started":1}
I20260619 14:41:19.887705 19009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:19.882726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45247 (local address 127.0.98.1:52273)
0619 14:41:19.883078 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:19.883092 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:19.883214 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:19.883465 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:19.883468 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:19.883501 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:41:19.883668 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:19.883674 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:19.884659 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:19.884663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:19.886260 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:19.886273 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:19.887158 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:19.887162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:19.887294 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:19.887338 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 14:41:19.887412 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 14:41:19.887520 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":182,"spinlock_wait_cycles":27904,"thread_start_us":85,"threads_started":1}
I20260619 14:41:19.888398 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e4e82aff6524ea8be3835e41516024d" candidate_uuid: "7b0b3627e30d4a15b1fe9ff2699d2e43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c725233c7f24cd78388a0209f5ab49c" is_pre_election: true
I20260619 14:41:19.888401 18803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e4e82aff6524ea8be3835e41516024d" candidate_uuid: "7b0b3627e30d4a15b1fe9ff2699d2e43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82a5d299d90a4840bd60371f7c9e3747" is_pre_election: true
I20260619 14:41:19.888675 18803 raft_consensus.cc:2454] T 7e4e82aff6524ea8be3835e41516024d P 82a5d299d90a4840bd60371f7c9e3747 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b0b3627e30d4a15b1fe9ff2699d2e43 in term 0.
I20260619 14:41:19.888674 18937 raft_consensus.cc:2454] T 7e4e82aff6524ea8be3835e41516024d P 0c725233c7f24cd78388a0209f5ab49c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b0b3627e30d4a15b1fe9ff2699d2e43 in term 0.
I20260619 14:41:19.889164 18605 leader_election.cc:302] T 7e4e82aff6524ea8be3835e41516024d P 7b0b3627e30d4a15b1fe9ff2699d2e43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c725233c7f24cd78388a0209f5ab49c, 7b0b3627e30d4a15b1fe9ff2699d2e43; no voters:
I20260619 14:41:19.889447 19005 raft_consensus.cc:2781] T 7e4e82aff6524ea8be3835e41516024d P 7b0b3627e30d4a15b1fe9ff2699d2e43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:41:19.889528 19005 raft_consensus.cc:482] T 7e4e82aff6524ea8be3835e41516024d P 7b0b3627e30d4a15b1fe9ff2699d2e43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:41:19.889570 19005 raft_consensus.cc:3037] T 7e4e82aff6524ea8be3835e41516024d P 7b0b3627e30d4a15b1fe9ff2699d2e43 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:41:19.890563 19005 raft_consensus.cc:504] T 7e4e82aff6524ea8be3835e41516024d P 7b0b3627e30d4a15b1fe9ff2699d2e43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a5d299d90a4840bd60371f7c9e3747" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45247 } } peers { permanent_uuid: "7b0b3627e30d4a15b1fe9ff2699d2e43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43187 } } peers { permanent_uuid: "0c725233c7f24cd78388a0209f5ab49c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45777 } }
I20260619 14:41:19.890887 19005 leader_election.cc:288] T 7e4e82aff6524ea8be3835e41516024d P 7b0b3627e30d4a15b1fe9ff2699d2e43 [CANDIDATE]: Term 1 election: Requested vote from peers 82a5d299d90a4840bd60371f7c9e3747 (127.0.98.2:45247), 0c725233c7f24cd78388a0209f5ab49c (127.0.98.3:45777)
I20260619 14:41:19.891208 18803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e4e82aff6524ea8be3835e41516024d" candidate_uuid: "7b0b3627e30d4a15b1fe9ff2699d2e43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82a5d299d90a4840bd60371f7c9e3747"
I20260619 14:41:19.891224 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e4e82aff6524ea8be3835e41516024d" candidate_uuid: "7b0b3627e30d4a15b1fe9ff2699d2e43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c725233c7f24cd78388a0209f5ab49c"
I20260619 14:41:19.891319 18803 raft_consensus.cc:3037] T 7e4e82aff6524ea8be3835e41516024d P 82a5d299d90a4840bd60371f7c9e3747 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:41:19.891325 18937 raft_consensus.cc:3037] T 7e4e82aff6524ea8be3835e41516024d P 0c725233c7f24cd78388a0209f5ab49c [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:41:19.892295 18803 raft_consensus.cc:2454] T 7e4e82aff6524ea8be3835e41516024d P 82a5d299d90a4840bd60371f7c9e3747 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b0b3627e30d4a15b1fe9ff2699d2e43 in term 1.
I20260619 14:41:19.892318 18937 raft_consensus.cc:2454] T 7e4e82aff6524ea8be3835e41516024d P 0c725233c7f24cd78388a0209f5ab49c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b0b3627e30d4a15b1fe9ff2699d2e43 in term 1.
I20260619 14:41:19.892580 18603 leader_election.cc:302] T 7e4e82aff6524ea8be3835e41516024d P 7b0b3627e30d4a15b1fe9ff2699d2e43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b0b3627e30d4a15b1fe9ff2699d2e43, 82a5d299d90a4840bd60371f7c9e3747; no voters:
I20260619 14:41:19.892774 19005 raft_consensus.cc:2781] T 7e4e82aff6524ea8be3835e41516024d P 7b0b3627e30d4a15b1fe9ff2699d2e43 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:41:19.892976 19005 raft_consensus.cc:686] T 7e4e82aff6524ea8be3835e41516024d P 7b0b3627e30d4a15b1fe9ff2699d2e43 [term 1 LEADER]: Becoming Leader. State: Replica: 7b0b3627e30d4a15b1fe9ff2699d2e43, State: Running, Role: LEADER
I20260619 14:41:19.893316 19005 consensus_queue.cc:237] T 7e4e82aff6524ea8be3835e41516024d P 7b0b3627e30d4a15b1fe9ff2699d2e43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a5d299d90a4840bd60371f7c9e3747" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45247 } } peers { permanent_uuid: "7b0b3627e30d4a15b1fe9ff2699d2e43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43187 } } peers { permanent_uuid: "0c725233c7f24cd78388a0209f5ab49c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45777 } }
I20260619 14:41:19.896373 18524 catalog_manager.cc:5495] T 7e4e82aff6524ea8be3835e41516024d P 7b0b3627e30d4a15b1fe9ff2699d2e43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b0b3627e30d4a15b1fe9ff2699d2e43 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "7b0b3627e30d4a15b1fe9ff2699d2e43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82a5d299d90a4840bd60371f7c9e3747" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b0b3627e30d4a15b1fe9ff2699d2e43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c725233c7f24cd78388a0209f5ab49c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45777 } health_report { overall_health: UNKNOWN } } }
I20260619 14:41:19.926117 18998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:19.922924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47698 (local address 127.0.98.1:43187)
0619 14:41:19.923040 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:19.923044 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:19.923058 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:41:19.923159 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:19.923161 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:19.923219 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:19.923306 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:41:19.923502 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:19.923999 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:19.924789 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:19.925009 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:19.925086 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:19.925322 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:41:19.925758 (+ 436us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:41:19.925867 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:19.925941 (+ 74us) server_negotiation.cc:299] Negotiation successful
0619 14:41:19.925973 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 14:41:19.931838 18937 raft_consensus.cc:1261] T 7e4e82aff6524ea8be3835e41516024d P 0c725233c7f24cd78388a0209f5ab49c [term 1 FOLLOWER]: Refusing update from remote peer 7b0b3627e30d4a15b1fe9ff2699d2e43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:41:19.931838 18803 raft_consensus.cc:1261] T 7e4e82aff6524ea8be3835e41516024d P 82a5d299d90a4840bd60371f7c9e3747 [term 1 FOLLOWER]: Refusing update from remote peer 7b0b3627e30d4a15b1fe9ff2699d2e43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:41:19.932677 19011 consensus_queue.cc:1035] T 7e4e82aff6524ea8be3835e41516024d P 7b0b3627e30d4a15b1fe9ff2699d2e43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c725233c7f24cd78388a0209f5ab49c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:41:19.932888 19005 consensus_queue.cc:1035] T 7e4e82aff6524ea8be3835e41516024d P 7b0b3627e30d4a15b1fe9ff2699d2e43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82a5d299d90a4840bd60371f7c9e3747" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:41:19.942770 19016 mvcc.cc:205] Tried to move back new op lower bound from 7298580807393652736 to 7298580807245680640. Current Snapshot: MvccSnapshot[applied={T|T < 7298580807393652736}]
I20260619 14:41:19.948381 18522 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:41:19.948583 18522 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:41:19.951059 18522 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0a39a240789e48daa1ac66aafd70e7fd: Sending DeleteTablet for 3 replicas of tablet 7e4e82aff6524ea8be3835e41516024d
I20260619 14:41:19.951894 18783 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e4e82aff6524ea8be3835e41516024d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:41:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52864
I20260619 14:41:19.951913 18917 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e4e82aff6524ea8be3835e41516024d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:41:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42444
I20260619 14:41:19.951924 18649 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e4e82aff6524ea8be3835e41516024d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:41:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47684
I20260619 14:41:19.952399 19023 tablet_replica.cc:330] T 7e4e82aff6524ea8be3835e41516024d P 7b0b3627e30d4a15b1fe9ff2699d2e43: stopping tablet replica
I20260619 14:41:19.952606 19024 tablet_replica.cc:330] T 7e4e82aff6524ea8be3835e41516024d P 82a5d299d90a4840bd60371f7c9e3747: stopping tablet replica
I20260619 14:41:19.952989 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 18586
I20260619 14:41:19.953686 19025 tablet_replica.cc:330] T 7e4e82aff6524ea8be3835e41516024d P 0c725233c7f24cd78388a0209f5ab49c: stopping tablet replica
I20260619 14:41:19.953719 19024 raft_consensus.cc:2229] T 7e4e82aff6524ea8be3835e41516024d P 82a5d299d90a4840bd60371f7c9e3747 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:41:19.953986 19025 raft_consensus.cc:2229] T 7e4e82aff6524ea8be3835e41516024d P 0c725233c7f24cd78388a0209f5ab49c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:41:19.954063 19024 raft_consensus.cc:2258] T 7e4e82aff6524ea8be3835e41516024d P 82a5d299d90a4840bd60371f7c9e3747 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:41:19.954233 19025 raft_consensus.cc:2258] T 7e4e82aff6524ea8be3835e41516024d P 0c725233c7f24cd78388a0209f5ab49c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:41:19.956233 19025 ts_tablet_manager.cc:1905] T 7e4e82aff6524ea8be3835e41516024d P 0c725233c7f24cd78388a0209f5ab49c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:41:19.956410 19024 ts_tablet_manager.cc:1905] T 7e4e82aff6524ea8be3835e41516024d P 82a5d299d90a4840bd60371f7c9e3747: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:41:19.958984 19025 ts_tablet_manager.cc:1918] T 7e4e82aff6524ea8be3835e41516024d P 0c725233c7f24cd78388a0209f5ab49c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:41:19.958984 19024 ts_tablet_manager.cc:1918] T 7e4e82aff6524ea8be3835e41516024d P 82a5d299d90a4840bd60371f7c9e3747: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:41:19.959088 19024 log.cc:1196] T 7e4e82aff6524ea8be3835e41516024d P 82a5d299d90a4840bd60371f7c9e3747: Deleting WAL directory at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/ts-1/wal/wals/7e4e82aff6524ea8be3835e41516024d
I20260619 14:41:19.959088 19025 log.cc:1196] T 7e4e82aff6524ea8be3835e41516024d P 0c725233c7f24cd78388a0209f5ab49c: Deleting WAL directory at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781879837648036-392-0/minicluster-data/ts-2/wal/wals/7e4e82aff6524ea8be3835e41516024d
I20260619 14:41:19.959399 19024 ts_tablet_manager.cc:1939] T 7e4e82aff6524ea8be3835e41516024d P 82a5d299d90a4840bd60371f7c9e3747: Deleting consensus metadata
I20260619 14:41:19.959399 19025 ts_tablet_manager.cc:1939] T 7e4e82aff6524ea8be3835e41516024d P 0c725233c7f24cd78388a0209f5ab49c: Deleting consensus metadata
W20260619 14:41:19.959530 18510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43187 (error 108)
I20260619 14:41:19.959820 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 18720
I20260619 14:41:19.961069 18997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:19.960807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43187 (local address 127.0.0.1:47704)
0619 14:41:19.960921 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:19.960991 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260619 14:41:19.961252 18510 catalog_manager.cc:4584] TS 7b0b3627e30d4a15b1fe9ff2699d2e43 (127.0.98.1:43187): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7e4e82aff6524ea8be3835e41516024d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43187: connect: Connection refused (error 111)
I20260619 14:41:19.964955 18511 catalog_manager.cc:4841] TS 0c725233c7f24cd78388a0209f5ab49c (127.0.98.3:45777): tablet 7e4e82aff6524ea8be3835e41516024d (table test-table [id=bf092b74f4fe44fbbae1f5432c3396af]) successfully deleted
I20260619 14:41:19.965596 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 18854
I20260619 14:41:19.967342 18997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:19.965700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45247 (local address 127.0.0.1:52870)
0619 14:41:19.967206 (+ 1506us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:19.967264 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:45247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1417}
I20260619 14:41:19.970994 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 18492
2026-06-19T14:41:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2166 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-19T14:41:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:41:19Z Disabled control of system clock
I20260619 14:41:19.999414 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38081
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38081
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 14:41:20.107489 19030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:41:20.107771 19030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:41:20.107805 19030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:41:20.111800 19030 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:41:20.111872 19030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:41:20.111886 19030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:41:20.111902 19030 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:41:20.111915 19030 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:41:20.116876 19030 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38081
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:41:20.118338 19030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:41:20.120195 19030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:41:20.128427 19035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:20.128520 19036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:20.128572 19030 server_base.cc:1034] running on GCE node
W20260619 14:41:20.128454 19038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:20.129281 19030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:41:20.130383 19030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:41:20.131563 19030 hybrid_clock.cc:648] HybridClock initialized: now 1781880080131542 us; error 35 us; skew 500 ppm
I20260619 14:41:20.133504 19030 webserver.cc:460] Webserver started at http://127.0.98.62:40295/ using document root <none> and password file <none>
I20260619 14:41:20.134115 19030 fs_manager.cc:359] Metadata directory not provided
I20260619 14:41:20.134188 19030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:41:20.134392 19030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:41:20.136449 19030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/master-0/data/instance:
uuid: "3542f6461b4541d8ba6092203af44c29"
format_stamp: "Formatted at 2026-06-19 14:41:20 on dist-test-slave-32hj"
server_key: "2f433e013152c8ae426056095a699564"
server_key_iv: "ad7e66e5c6e4d133017f997b4092e2e2"
server_key_version: "encryptionkey@0"
I20260619 14:41:20.137015 19030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/master-0/wal/instance:
uuid: "3542f6461b4541d8ba6092203af44c29"
format_stamp: "Formatted at 2026-06-19 14:41:20 on dist-test-slave-32hj"
server_key: "2f433e013152c8ae426056095a699564"
server_key_iv: "ad7e66e5c6e4d133017f997b4092e2e2"
server_key_version: "encryptionkey@0"
I20260619 14:41:20.140791 19030 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260619 14:41:20.143354 19044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:20.144804 19030 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 14:41:20.144930 19030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/master-0/wal
uuid: "3542f6461b4541d8ba6092203af44c29"
format_stamp: "Formatted at 2026-06-19 14:41:20 on dist-test-slave-32hj"
server_key: "2f433e013152c8ae426056095a699564"
server_key_iv: "ad7e66e5c6e4d133017f997b4092e2e2"
server_key_version: "encryptionkey@0"
I20260619 14:41:20.145043 19030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:41:20.168565 19030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:41:20.169293 19030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:41:20.169476 19030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:41:20.176041 19030 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38081
I20260619 14:41:20.176064 19096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38081 every 8 connection(s)
I20260619 14:41:20.176999 19030 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:41:20.179981 19097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:41:20.184943 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 19030
I20260619 14:41:20.185065 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/master-0/wal/instance
I20260619 14:41:20.185303 392 external_mini_cluster.cc:1428] Setting key 0569142b1b78e284684a7c237043bf4e
I20260619 14:41:20.185295 19097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3542f6461b4541d8ba6092203af44c29: Bootstrap starting.
I20260619 14:41:20.187719 19097 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3542f6461b4541d8ba6092203af44c29: Neither blocks nor log segments found. Creating new log.
I20260619 14:41:20.188426 19097 log.cc:824] T 00000000000000000000000000000000 P 3542f6461b4541d8ba6092203af44c29: Log is configured to *not* fsync() on all Append() calls
I20260619 14:41:20.190131 19097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3542f6461b4541d8ba6092203af44c29: No bootstrap required, opened a new log
I20260619 14:41:20.193626 19097 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3542f6461b4541d8ba6092203af44c29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3542f6461b4541d8ba6092203af44c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38081 } }
I20260619 14:41:20.193836 19097 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3542f6461b4541d8ba6092203af44c29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:41:20.193866 19100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:20.186534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35094 (local address 127.0.98.62:38081)
0619 14:41:20.187157 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:20.187170 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:20.187204 (+ 34us) server_negotiation.cc:407] Connection header received
0619 14:41:20.187802 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:20.187821 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:20.188234 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:20.188540 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:20.188975 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:20.189900 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:20.190742 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:20.191213 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:20.191358 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:20.192198 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:20.192203 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:20.192216 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:41:20.192592 (+ 376us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:20.192645 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:41:20.192664 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:20.192771 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:20.192900 (+ 129us) server_negotiation.cc:299] Negotiation successful
0619 14:41:20.193142 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":127,"threads_started":1}
I20260619 14:41:20.193890 19097 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3542f6461b4541d8ba6092203af44c29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3542f6461b4541d8ba6092203af44c29, State: Initialized, Role: FOLLOWER
I20260619 14:41:20.194382 19097 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3542f6461b4541d8ba6092203af44c29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3542f6461b4541d8ba6092203af44c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38081 } }
I20260619 14:41:20.194533 19097 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3542f6461b4541d8ba6092203af44c29 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:41:20.194593 19097 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3542f6461b4541d8ba6092203af44c29 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:41:20.194682 19097 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3542f6461b4541d8ba6092203af44c29 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:41:20.195505 19097 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3542f6461b4541d8ba6092203af44c29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3542f6461b4541d8ba6092203af44c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38081 } }
I20260619 14:41:20.195828 19097 leader_election.cc:302] T 00000000000000000000000000000000 P 3542f6461b4541d8ba6092203af44c29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3542f6461b4541d8ba6092203af44c29; no voters:
I20260619 14:41:20.196121 19097 leader_election.cc:288] T 00000000000000000000000000000000 P 3542f6461b4541d8ba6092203af44c29 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:41:20.196213 19102 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3542f6461b4541d8ba6092203af44c29 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:41:20.196440 19102 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3542f6461b4541d8ba6092203af44c29 [term 1 LEADER]: Becoming Leader. State: Replica: 3542f6461b4541d8ba6092203af44c29, State: Running, Role: LEADER
I20260619 14:41:20.196712 19102 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3542f6461b4541d8ba6092203af44c29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3542f6461b4541d8ba6092203af44c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38081 } }
I20260619 14:41:20.197259 19097 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3542f6461b4541d8ba6092203af44c29 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:41:20.198226 19103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3542f6461b4541d8ba6092203af44c29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3542f6461b4541d8ba6092203af44c29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3542f6461b4541d8ba6092203af44c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38081 } } }
I20260619 14:41:20.198345 19103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3542f6461b4541d8ba6092203af44c29 [sys.catalog]: This master's current role is: LEADER
I20260619 14:41:20.198587 19104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3542f6461b4541d8ba6092203af44c29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3542f6461b4541d8ba6092203af44c29. Latest consensus state: current_term: 1 leader_uuid: "3542f6461b4541d8ba6092203af44c29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3542f6461b4541d8ba6092203af44c29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38081 } } }
I20260619 14:41:20.198683 19104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3542f6461b4541d8ba6092203af44c29 [sys.catalog]: This master's current role is: LEADER
I20260619 14:41:20.198762 19111 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:41:20.201689 19111 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:41:20.210134 19111 catalog_manager.cc:1347] Generated new cluster ID: 5e86c217cac74461a8e3774a03a6f169
I20260619 14:41:20.210211 19111 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:41:20.217211 19111 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:41:20.218410 19111 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:41:20.224097 19111 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3542f6461b4541d8ba6092203af44c29: Generated new TSK 0
I20260619 14:41:20.224767 19111 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:41:20.234819 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38081
--builtin_ntp_servers=127.0.98.20:45131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 14:41:20.345399 19121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:41:20.345664 19121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:41:20.345702 19121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:41:20.349746 19121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:41:20.349820 19121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:41:20.349887 19121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:41:20.354712 19121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:41:20.355774 19121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:41:20.357151 19121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:41:20.366391 19129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:20.366417 19127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:20.366562 19121 server_base.cc:1034] running on GCE node
W20260619 14:41:20.366415 19126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:20.367081 19121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:41:20.367688 19121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:41:20.368817 19121 hybrid_clock.cc:648] HybridClock initialized: now 1781880080368800 us; error 34 us; skew 500 ppm
I20260619 14:41:20.370615 19121 webserver.cc:460] Webserver started at http://127.0.98.1:33199/ using document root <none> and password file <none>
I20260619 14:41:20.371171 19121 fs_manager.cc:359] Metadata directory not provided
I20260619 14:41:20.371234 19121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:41:20.371443 19121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:41:20.373518 19121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/ts-0/data/instance:
uuid: "fe80663143b84f279b195cc24f47184a"
format_stamp: "Formatted at 2026-06-19 14:41:20 on dist-test-slave-32hj"
server_key: "e320339fbfd45acdb0cf1c997b1baaf6"
server_key_iv: "d93fd80085ea530261e78c0c652c2a49"
server_key_version: "encryptionkey@0"
I20260619 14:41:20.374033 19121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/ts-0/wal/instance:
uuid: "fe80663143b84f279b195cc24f47184a"
format_stamp: "Formatted at 2026-06-19 14:41:20 on dist-test-slave-32hj"
server_key: "e320339fbfd45acdb0cf1c997b1baaf6"
server_key_iv: "d93fd80085ea530261e78c0c652c2a49"
server_key_version: "encryptionkey@0"
I20260619 14:41:20.377660 19121 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 14:41:20.380079 19135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:20.381361 19121 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 14:41:20.381466 19121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/ts-0/wal
uuid: "fe80663143b84f279b195cc24f47184a"
format_stamp: "Formatted at 2026-06-19 14:41:20 on dist-test-slave-32hj"
server_key: "e320339fbfd45acdb0cf1c997b1baaf6"
server_key_iv: "d93fd80085ea530261e78c0c652c2a49"
server_key_version: "encryptionkey@0"
I20260619 14:41:20.381564 19121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:41:20.396193 19121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:41:20.396847 19121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:41:20.397007 19121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:41:20.397598 19121 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:41:20.398479 19121 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:41:20.398526 19121 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:20.398576 19121 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:41:20.398597 19121 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:20.406589 19121 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44453
I20260619 14:41:20.406616 19248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44453 every 8 connection(s)
I20260619 14:41:20.407433 19121 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:41:20.410933 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 19121
I20260619 14:41:20.411065 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:41:20.411377 392 external_mini_cluster.cc:1428] Setting key c90a19b595fe70e79ae536b3513180dc
I20260619 14:41:20.413447 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38081
--builtin_ntp_servers=127.0.98.20:45131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 14:41:20.417198 19100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:20.409128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53323 (local address 127.0.98.62:38081)
0619 14:41:20.409265 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:20.409270 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:20.410306 (+ 1036us) server_negotiation.cc:407] Connection header received
0619 14:41:20.412092 (+ 1786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:20.412095 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:20.412148 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:20.412233 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:20.413360 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:20.413970 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:20.415300 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:20.415609 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:20.415694 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:20.416714 (+ 1020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:20.416734 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:20.416736 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:41:20.416751 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:20.416775 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:41:20.416785 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:20.416851 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:20.416993 (+ 142us) server_negotiation.cc:299] Negotiation successful
0619 14:41:20.417056 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 14:41:20.418083 19251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:20.409544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38081 (local address 127.0.98.1:53323)
0619 14:41:20.410161 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:20.410201 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:20.411871 (+ 1670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:20.412425 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:20.412432 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:20.412518 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:41:20.413212 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:20.413233 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:20.414091 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:20.414094 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:20.415181 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:20.415191 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:20.415747 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:20.415750 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:20.415896 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:20.416464 (+ 568us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:41:20.416466 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:41:20.416507 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:41:20.416571 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:41:20.416864 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:41:20.416873 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:41:20.416879 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 14:41:20.416988 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 14:41:20.417281 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":105,"threads_started":1}
I20260619 14:41:20.419250 19249 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38081
I20260619 14:41:20.419474 19249 heartbeater.cc:464] Registering TS with master...
I20260619 14:41:20.420003 19249 heartbeater.cc:511] Master 127.0.98.62:38081 requested a full tablet report, sending...
I20260619 14:41:20.421623 19061 ts_manager.cc:194] Registered new tserver with Master: fe80663143b84f279b195cc24f47184a (127.0.98.1:44453)
I20260619 14:41:20.423218 19061 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:53323
W20260619 14:41:20.521720 19252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:41:20.521987 19252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:41:20.522023 19252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:41:20.526000 19252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:41:20.526051 19252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:41:20.526105 19252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:41:20.530829 19252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:41:20.531839 19252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:41:20.533190 19252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:41:20.542680 19257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:20.542709 19258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:20.542714 19252 server_base.cc:1034] running on GCE node
W20260619 14:41:20.542732 19260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:20.543327 19252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:41:20.543977 19252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:41:20.545143 19252 hybrid_clock.cc:648] HybridClock initialized: now 1781880080545122 us; error 34 us; skew 500 ppm
I20260619 14:41:20.547178 19252 webserver.cc:460] Webserver started at http://127.0.98.2:39899/ using document root <none> and password file <none>
I20260619 14:41:20.547731 19252 fs_manager.cc:359] Metadata directory not provided
I20260619 14:41:20.547798 19252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:41:20.548043 19252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:41:20.550238 19252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/ts-1/data/instance:
uuid: "c3871466d01243979ed8454e10569ebc"
format_stamp: "Formatted at 2026-06-19 14:41:20 on dist-test-slave-32hj"
server_key: "0ddb7a37b7346157a4be0537f3062fd2"
server_key_iv: "8cc78a9e5d9a2430953864ddce81cde0"
server_key_version: "encryptionkey@0"
I20260619 14:41:20.550781 19252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c3871466d01243979ed8454e10569ebc"
format_stamp: "Formatted at 2026-06-19 14:41:20 on dist-test-slave-32hj"
server_key: "0ddb7a37b7346157a4be0537f3062fd2"
server_key_iv: "8cc78a9e5d9a2430953864ddce81cde0"
server_key_version: "encryptionkey@0"
I20260619 14:41:20.554479 19252 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:41:20.556988 19266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:20.558158 19252 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:41:20.558287 19252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/ts-1/wal
uuid: "c3871466d01243979ed8454e10569ebc"
format_stamp: "Formatted at 2026-06-19 14:41:20 on dist-test-slave-32hj"
server_key: "0ddb7a37b7346157a4be0537f3062fd2"
server_key_iv: "8cc78a9e5d9a2430953864ddce81cde0"
server_key_version: "encryptionkey@0"
I20260619 14:41:20.558385 19252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:41:20.576529 19252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:41:20.577268 19252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:41:20.577452 19252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:41:20.578023 19252 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:41:20.578961 19252 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:41:20.579007 19252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:20.579059 19252 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:41:20.579088 19252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:20.587495 19252 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41581
I20260619 14:41:20.587528 19379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41581 every 8 connection(s)
I20260619 14:41:20.588446 19252 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:41:20.589681 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 19252
I20260619 14:41:20.589802 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:41:20.590106 392 external_mini_cluster.cc:1428] Setting key 27f1501d9d1e4b7d8e942f1dd92c05f8
I20260619 14:41:20.592049 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38081
--builtin_ntp_servers=127.0.98.20:45131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 14:41:20.597966 19100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:20.590275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60229 (local address 127.0.98.62:38081)
0619 14:41:20.590430 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:20.590435 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:20.591659 (+ 1224us) server_negotiation.cc:407] Connection header received
0619 14:41:20.593067 (+ 1408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:20.593070 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:20.593127 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:20.593213 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:20.594325 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:20.594851 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:20.595959 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:20.596177 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:20.596262 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:20.597387 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:20.597392 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:20.597395 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:41:20.597411 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:20.597434 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:41:20.597442 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:20.597503 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:20.597689 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 14:41:20.597753 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 14:41:20.598762 19382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:20.590744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38081 (local address 127.0.98.2:60229)
0619 14:41:20.591510 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:20.591564 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:20.592859 (+ 1295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:20.593400 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:20.593409 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:20.593522 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:41:20.594188 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:20.594206 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:20.594964 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:20.594967 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:20.595849 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:20.595857 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:20.596309 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:20.596313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:20.596477 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:20.597131 (+ 654us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:41:20.597132 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:41:20.597174 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:41:20.597239 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:41:20.597526 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:41:20.597535 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:41:20.597542 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 14:41:20.597665 (+ 123us) client_negotiation.cc:240] Negotiation successful
0619 14:41:20.597948 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":104,"threads_started":1}
I20260619 14:41:20.599895 19380 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38081
I20260619 14:41:20.600127 19380 heartbeater.cc:464] Registering TS with master...
I20260619 14:41:20.600636 19380 heartbeater.cc:511] Master 127.0.98.62:38081 requested a full tablet report, sending...
I20260619 14:41:20.601649 19061 ts_manager.cc:194] Registered new tserver with Master: c3871466d01243979ed8454e10569ebc (127.0.98.2:41581)
I20260619 14:41:20.602371 19061 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:60229
W20260619 14:41:20.699961 19383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:41:20.700212 19383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:41:20.700249 19383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:41:20.704216 19383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:41:20.704274 19383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:41:20.704327 19383 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:41:20.708972 19383 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19383
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:41:20.709936 19383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:41:20.711174 19383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:41:20.720057 19391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:20.720118 19388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:20.720171 19389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:20.720227 19383 server_base.cc:1034] running on GCE node
I20260619 14:41:20.720783 19383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:41:20.721388 19383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:41:20.722582 19383 hybrid_clock.cc:648] HybridClock initialized: now 1781880080722542 us; error 43 us; skew 500 ppm
I20260619 14:41:20.724460 19383 webserver.cc:460] Webserver started at http://127.0.98.3:43485/ using document root <none> and password file <none>
I20260619 14:41:20.725056 19383 fs_manager.cc:359] Metadata directory not provided
I20260619 14:41:20.725127 19383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:41:20.725337 19383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:41:20.727473 19383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/ts-2/data/instance:
uuid: "ebd15815082e48c996faf4462d0f18c8"
format_stamp: "Formatted at 2026-06-19 14:41:20 on dist-test-slave-32hj"
server_key: "9257f207a44e1d929469959074bc25c4"
server_key_iv: "a23d950c1b4dea863f9a287492b00e2c"
server_key_version: "encryptionkey@0"
I20260619 14:41:20.728022 19383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ebd15815082e48c996faf4462d0f18c8"
format_stamp: "Formatted at 2026-06-19 14:41:20 on dist-test-slave-32hj"
server_key: "9257f207a44e1d929469959074bc25c4"
server_key_iv: "a23d950c1b4dea863f9a287492b00e2c"
server_key_version: "encryptionkey@0"
I20260619 14:41:20.731551 19383 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 14:41:20.733915 19397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:20.735183 19383 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:41:20.735289 19383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/ts-2/wal
uuid: "ebd15815082e48c996faf4462d0f18c8"
format_stamp: "Formatted at 2026-06-19 14:41:20 on dist-test-slave-32hj"
server_key: "9257f207a44e1d929469959074bc25c4"
server_key_iv: "a23d950c1b4dea863f9a287492b00e2c"
server_key_version: "encryptionkey@0"
I20260619 14:41:20.735411 19383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:41:20.757355 19383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:41:20.758013 19383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:41:20.758163 19383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:41:20.758689 19383 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:41:20.759541 19383 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:41:20.759584 19383 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:20.759622 19383 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:41:20.759641 19383 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:20.767688 19383 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34151
I20260619 14:41:20.767742 19510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34151 every 8 connection(s)
I20260619 14:41:20.768601 19383 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:41:20.777421 19100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:20.770342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53517 (local address 127.0.98.62:38081)
0619 14:41:20.770498 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:20.770503 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:20.771619 (+ 1116us) server_negotiation.cc:407] Connection header received
0619 14:41:20.772959 (+ 1340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:20.772962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:20.773008 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:20.773084 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:20.773978 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:20.774535 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:20.775659 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:20.775874 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:20.775946 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:20.776952 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:20.776955 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:20.776957 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:41:20.776969 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:20.776988 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:41:20.776995 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:20.777054 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:20.777231 (+ 177us) server_negotiation.cc:299] Negotiation successful
0619 14:41:20.777281 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 14:41:20.778223 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 19383
I20260619 14:41:20.778342 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:41:20.778373 19513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:20.770754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38081 (local address 127.0.98.3:53517)
0619 14:41:20.771447 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:20.771488 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:20.772770 (+ 1282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:20.773239 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:20.773245 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:20.773330 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:41:20.773834 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:20.773847 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:20.774669 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:20.774672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:20.775562 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:20.775571 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:20.775985 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:20.775987 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:20.776131 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:20.776662 (+ 531us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:41:20.776663 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:41:20.776702 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:41:20.776814 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:41:20.777091 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:41:20.777100 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:41:20.777104 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 14:41:20.777238 (+ 134us) client_negotiation.cc:240] Negotiation successful
0619 14:41:20.777459 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":122,"threads_started":1}
I20260619 14:41:20.778641 392 external_mini_cluster.cc:1428] Setting key b87dd82d8e6437b8be43bfba5e960fee
I20260619 14:41:20.779534 19511 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38081
I20260619 14:41:20.779762 19511 heartbeater.cc:464] Registering TS with master...
I20260619 14:41:20.780267 19511 heartbeater.cc:511] Master 127.0.98.62:38081 requested a full tablet report, sending...
I20260619 14:41:20.781199 19061 ts_manager.cc:194] Registered new tserver with Master: ebd15815082e48c996faf4462d0f18c8 (127.0.98.3:34151)
I20260619 14:41:20.781759 19061 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:53517
I20260619 14:41:20.792301 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:41:20.798056 19100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:20.794187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35096 (local address 127.0.98.62:38081)
0619 14:41:20.794304 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:20.794308 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:20.794447 (+ 139us) server_negotiation.cc:407] Connection header received
0619 14:41:20.794545 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:20.794547 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:20.794587 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:20.794651 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:20.794982 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:20.795499 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:20.796410 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:20.796631 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:20.796746 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:20.797595 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:20.797600 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:20.797603 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:41:20.797616 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:20.797635 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:41:20.797643 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:20.797741 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:20.797902 (+ 161us) server_negotiation.cc:299] Negotiation successful
0619 14:41:20.797937 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 14:41:20.801100 19061 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35096:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:41:20.804016 19061 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:41:20.818363 19526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:20.813382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41581 (local address 127.0.0.1:48302)
0619 14:41:20.813771 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:20.813784 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:20.813875 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:20.814715 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:20.814718 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:20.814735 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:41:20.814812 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:20.814817 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:20.816056 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:20.816059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:20.817086 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:20.817095 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:20.817858 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:20.817861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:20.817960 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:20.817996 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 14:41:20.818093 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 14:41:20.818210 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":29,"thread_start_us":126,"threads_started":1}
I20260619 14:41:20.818372 19527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:20.813814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48302 (local address 127.0.98.2:41581)
0619 14:41:20.814286 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:20.814302 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:20.814323 (+ 21us) server_negotiation.cc:407] Connection header received
0619 14:41:20.814396 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:20.814401 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:20.814560 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:20.814721 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:41:20.814915 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:20.815960 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:20.817227 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:20.817748 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:20.817845 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:20.818033 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:20.818119 (+ 86us) server_negotiation.cc:299] Negotiation successful
0619 14:41:20.818221 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":113,"threads_started":1}
I20260619 14:41:20.818581 19522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:20.812591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34151 (local address 127.0.0.1:46926)
0619 14:41:20.813315 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:20.813363 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:20.813514 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:20.814231 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:20.814237 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:20.814271 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:41:20.814459 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:20.814465 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:20.815546 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:20.815550 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:20.816838 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:20.816849 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:20.818094 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:20.818098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:20.818234 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:20.818284 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 14:41:20.818349 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 14:41:20.818431 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"mutex_wait_us":104,"spinlock_wait_cycles":6784,"thread_start_us":130,"threads_started":1}
I20260619 14:41:20.818953 19523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:20.812597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44453 (local address 127.0.0.1:52262)
0619 14:41:20.813590 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:20.813602 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:20.813680 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:20.814231 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:20.814237 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:20.814273 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:41:20.814455 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:20.814463 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:20.815857 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:20.815861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:20.817025 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:20.817033 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:20.818114 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:20.818117 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:20.818250 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:20.818285 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 14:41:20.818349 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 14:41:20.818841 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":938,"mutex_wait_us":686,"spinlock_wait_cycles":8576,"thread_start_us":100,"threads_started":1}
I20260619 14:41:20.819054 19525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:20.812853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52262 (local address 127.0.98.1:44453)
0619 14:41:20.813259 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:20.813270 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:20.813761 (+ 491us) server_negotiation.cc:407] Connection header received
0619 14:41:20.813827 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:20.813831 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:20.813967 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:20.814078 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:41:20.814571 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:20.815746 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:20.817137 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:20.817851 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:20.817933 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:20.818069 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:20.818421 (+ 352us) server_negotiation.cc:299] Negotiation successful
0619 14:41:20.818916 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":104,"threads_started":1}
I20260619 14:41:20.819418 19524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:20.812781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46926 (local address 127.0.98.3:34151)
0619 14:41:20.813435 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:20.813445 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:20.813470 (+ 25us) server_negotiation.cc:407] Connection header received
0619 14:41:20.813722 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:20.813728 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:20.813933 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:20.814074 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:41:20.814597 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:20.815448 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:20.816967 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:20.817749 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:20.817847 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:20.818033 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:20.819161 (+ 1128us) server_negotiation.cc:299] Negotiation successful
0619 14:41:20.819269 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":169,"threads_started":1}
I20260619 14:41:20.821338 19314 tablet_service.cc:1467] Processing CreateTablet for tablet e0f4cffc43854a3991a47591cf873036 (DEFAULT_TABLE table=test-table [id=1a541c4cab114207886122955d205f6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:41:20.821446 19183 tablet_service.cc:1467] Processing CreateTablet for tablet e0f4cffc43854a3991a47591cf873036 (DEFAULT_TABLE table=test-table [id=1a541c4cab114207886122955d205f6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:41:20.822259 19314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0f4cffc43854a3991a47591cf873036. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:41:20.822259 19183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0f4cffc43854a3991a47591cf873036. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:41:20.826848 19445 tablet_service.cc:1467] Processing CreateTablet for tablet e0f4cffc43854a3991a47591cf873036 (DEFAULT_TABLE table=test-table [id=1a541c4cab114207886122955d205f6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:41:20.827937 19445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0f4cffc43854a3991a47591cf873036. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:41:20.828218 19528 tablet_bootstrap.cc:492] T e0f4cffc43854a3991a47591cf873036 P c3871466d01243979ed8454e10569ebc: Bootstrap starting.
I20260619 14:41:20.829828 19529 tablet_bootstrap.cc:492] T e0f4cffc43854a3991a47591cf873036 P fe80663143b84f279b195cc24f47184a: Bootstrap starting.
I20260619 14:41:20.830890 19528 tablet_bootstrap.cc:654] T e0f4cffc43854a3991a47591cf873036 P c3871466d01243979ed8454e10569ebc: Neither blocks nor log segments found. Creating new log.
I20260619 14:41:20.831741 19528 log.cc:824] T e0f4cffc43854a3991a47591cf873036 P c3871466d01243979ed8454e10569ebc: Log is configured to *not* fsync() on all Append() calls
I20260619 14:41:20.831895 19529 tablet_bootstrap.cc:654] T e0f4cffc43854a3991a47591cf873036 P fe80663143b84f279b195cc24f47184a: Neither blocks nor log segments found. Creating new log.
I20260619 14:41:20.832616 19529 log.cc:824] T e0f4cffc43854a3991a47591cf873036 P fe80663143b84f279b195cc24f47184a: Log is configured to *not* fsync() on all Append() calls
I20260619 14:41:20.833855 19528 tablet_bootstrap.cc:492] T e0f4cffc43854a3991a47591cf873036 P c3871466d01243979ed8454e10569ebc: No bootstrap required, opened a new log
I20260619 14:41:20.834055 19528 ts_tablet_manager.cc:1397] T e0f4cffc43854a3991a47591cf873036 P c3871466d01243979ed8454e10569ebc: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 14:41:20.834208 19532 tablet_bootstrap.cc:492] T e0f4cffc43854a3991a47591cf873036 P ebd15815082e48c996faf4462d0f18c8: Bootstrap starting.
I20260619 14:41:20.835024 19529 tablet_bootstrap.cc:492] T e0f4cffc43854a3991a47591cf873036 P fe80663143b84f279b195cc24f47184a: No bootstrap required, opened a new log
I20260619 14:41:20.835228 19529 ts_tablet_manager.cc:1397] T e0f4cffc43854a3991a47591cf873036 P fe80663143b84f279b195cc24f47184a: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260619 14:41:20.836268 19532 tablet_bootstrap.cc:654] T e0f4cffc43854a3991a47591cf873036 P ebd15815082e48c996faf4462d0f18c8: Neither blocks nor log segments found. Creating new log.
I20260619 14:41:20.836937 19532 log.cc:824] T e0f4cffc43854a3991a47591cf873036 P ebd15815082e48c996faf4462d0f18c8: Log is configured to *not* fsync() on all Append() calls
I20260619 14:41:20.838439 19532 tablet_bootstrap.cc:492] T e0f4cffc43854a3991a47591cf873036 P ebd15815082e48c996faf4462d0f18c8: No bootstrap required, opened a new log
I20260619 14:41:20.838601 19532 ts_tablet_manager.cc:1397] T e0f4cffc43854a3991a47591cf873036 P ebd15815082e48c996faf4462d0f18c8: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 14:41:20.838872 19528 raft_consensus.cc:348] T e0f4cffc43854a3991a47591cf873036 P c3871466d01243979ed8454e10569ebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd15815082e48c996faf4462d0f18c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34151 } } peers { permanent_uuid: "fe80663143b84f279b195cc24f47184a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44453 } } peers { permanent_uuid: "c3871466d01243979ed8454e10569ebc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41581 } }
I20260619 14:41:20.838872 19529 raft_consensus.cc:348] T e0f4cffc43854a3991a47591cf873036 P fe80663143b84f279b195cc24f47184a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd15815082e48c996faf4462d0f18c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34151 } } peers { permanent_uuid: "fe80663143b84f279b195cc24f47184a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44453 } } peers { permanent_uuid: "c3871466d01243979ed8454e10569ebc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41581 } }
I20260619 14:41:20.839141 19528 raft_consensus.cc:374] T e0f4cffc43854a3991a47591cf873036 P c3871466d01243979ed8454e10569ebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:41:20.839141 19529 raft_consensus.cc:374] T e0f4cffc43854a3991a47591cf873036 P fe80663143b84f279b195cc24f47184a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:41:20.839205 19528 raft_consensus.cc:729] T e0f4cffc43854a3991a47591cf873036 P c3871466d01243979ed8454e10569ebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3871466d01243979ed8454e10569ebc, State: Initialized, Role: FOLLOWER
I20260619 14:41:20.839236 19529 raft_consensus.cc:729] T e0f4cffc43854a3991a47591cf873036 P fe80663143b84f279b195cc24f47184a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe80663143b84f279b195cc24f47184a, State: Initialized, Role: FOLLOWER
I20260619 14:41:20.839824 19528 consensus_queue.cc:260] T e0f4cffc43854a3991a47591cf873036 P c3871466d01243979ed8454e10569ebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd15815082e48c996faf4462d0f18c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34151 } } peers { permanent_uuid: "fe80663143b84f279b195cc24f47184a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44453 } } peers { permanent_uuid: "c3871466d01243979ed8454e10569ebc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41581 } }
I20260619 14:41:20.839819 19529 consensus_queue.cc:260] T e0f4cffc43854a3991a47591cf873036 P fe80663143b84f279b195cc24f47184a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd15815082e48c996faf4462d0f18c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34151 } } peers { permanent_uuid: "fe80663143b84f279b195cc24f47184a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44453 } } peers { permanent_uuid: "c3871466d01243979ed8454e10569ebc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41581 } }
I20260619 14:41:20.840525 19380 heartbeater.cc:503] Master 127.0.98.62:38081 was elected leader, sending a full tablet report...
I20260619 14:41:20.840827 19249 heartbeater.cc:503] Master 127.0.98.62:38081 was elected leader, sending a full tablet report...
I20260619 14:41:20.840987 19528 ts_tablet_manager.cc:1428] T e0f4cffc43854a3991a47591cf873036 P c3871466d01243979ed8454e10569ebc: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260619 14:41:20.841639 19529 ts_tablet_manager.cc:1428] T e0f4cffc43854a3991a47591cf873036 P fe80663143b84f279b195cc24f47184a: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260619 14:41:20.842694 19532 raft_consensus.cc:348] T e0f4cffc43854a3991a47591cf873036 P ebd15815082e48c996faf4462d0f18c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd15815082e48c996faf4462d0f18c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34151 } } peers { permanent_uuid: "fe80663143b84f279b195cc24f47184a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44453 } } peers { permanent_uuid: "c3871466d01243979ed8454e10569ebc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41581 } }
I20260619 14:41:20.842978 19532 raft_consensus.cc:374] T e0f4cffc43854a3991a47591cf873036 P ebd15815082e48c996faf4462d0f18c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:41:20.843035 19532 raft_consensus.cc:729] T e0f4cffc43854a3991a47591cf873036 P ebd15815082e48c996faf4462d0f18c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebd15815082e48c996faf4462d0f18c8, State: Initialized, Role: FOLLOWER
I20260619 14:41:20.843483 19532 consensus_queue.cc:260] T e0f4cffc43854a3991a47591cf873036 P ebd15815082e48c996faf4462d0f18c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd15815082e48c996faf4462d0f18c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34151 } } peers { permanent_uuid: "fe80663143b84f279b195cc24f47184a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44453 } } peers { permanent_uuid: "c3871466d01243979ed8454e10569ebc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41581 } }
I20260619 14:41:20.844092 19511 heartbeater.cc:503] Master 127.0.98.62:38081 was elected leader, sending a full tablet report...
I20260619 14:41:20.845347 19532 ts_tablet_manager.cc:1428] T e0f4cffc43854a3991a47591cf873036 P ebd15815082e48c996faf4462d0f18c8: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
W20260619 14:41:20.908614 19250 tablet.cc:2369] T e0f4cffc43854a3991a47591cf873036 P fe80663143b84f279b195cc24f47184a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:41:21.019539 19512 tablet.cc:2369] T e0f4cffc43854a3991a47591cf873036 P ebd15815082e48c996faf4462d0f18c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:41:21.084321 19536 raft_consensus.cc:482] T e0f4cffc43854a3991a47591cf873036 P ebd15815082e48c996faf4462d0f18c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:41:21.084542 19536 raft_consensus.cc:504] T e0f4cffc43854a3991a47591cf873036 P ebd15815082e48c996faf4462d0f18c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd15815082e48c996faf4462d0f18c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34151 } } peers { permanent_uuid: "fe80663143b84f279b195cc24f47184a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44453 } } peers { permanent_uuid: "c3871466d01243979ed8454e10569ebc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41581 } }
I20260619 14:41:21.085820 19536 leader_election.cc:288] T e0f4cffc43854a3991a47591cf873036 P ebd15815082e48c996faf4462d0f18c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe80663143b84f279b195cc24f47184a (127.0.98.1:44453), c3871466d01243979ed8454e10569ebc (127.0.98.2:41581)
I20260619 14:41:21.089429 19513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:21.086056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44453 (local address 127.0.98.3:52589)
0619 14:41:21.086196 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:21.086212 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:21.086314 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:21.086600 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:21.086603 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:21.086624 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:41:21.086744 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:21.086751 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:21.087473 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:21.087476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:21.088268 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:21.088277 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:21.089022 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:21.089025 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:21.089122 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:21.089145 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 14:41:21.089212 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 14:41:21.089294 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 14:41:21.089442 19525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:21.086134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52589 (local address 127.0.98.1:44453)
0619 14:41:21.086250 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:21.086253 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:21.086269 (+ 16us) server_negotiation.cc:407] Connection header received
0619 14:41:21.086427 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:21.086430 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:21.086478 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:21.086556 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:41:21.086854 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:21.087371 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:21.088386 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:21.088906 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:21.089012 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:21.089076 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:21.089243 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 14:41:21.089335 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260619 14:41:21.089618 19381 tablet.cc:2369] T e0f4cffc43854a3991a47591cf873036 P c3871466d01243979ed8454e10569ebc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:41:21.089785 19527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:21.086507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39051 (local address 127.0.98.2:41581)
0619 14:41:21.086665 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:21.086670 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:21.086827 (+ 157us) server_negotiation.cc:407] Connection header received
0619 14:41:21.086960 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:21.086963 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:21.087007 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:21.087068 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:41:21.087326 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:21.087803 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:21.088848 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:21.089253 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:21.089348 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:21.089405 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:21.089598 (+ 193us) server_negotiation.cc:299] Negotiation successful
0619 14:41:21.089684 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 14:41:21.090173 19537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:21.086356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41581 (local address 127.0.98.3:39051)
0619 14:41:21.086747 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:21.086757 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:21.086846 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:21.087088 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:21.087091 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:21.087106 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:41:21.087224 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:21.087230 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:21.087916 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:21.087920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:21.088706 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:21.088715 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:21.089371 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:21.089374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:21.089467 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:21.089487 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 14:41:21.089949 (+ 462us) client_negotiation.cc:240] Negotiation successful
0619 14:41:21.090020 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":117,"threads_started":1}
I20260619 14:41:21.090327 19203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f4cffc43854a3991a47591cf873036" candidate_uuid: "ebd15815082e48c996faf4462d0f18c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe80663143b84f279b195cc24f47184a" is_pre_election: true
I20260619 14:41:21.090651 19203 raft_consensus.cc:2454] T e0f4cffc43854a3991a47591cf873036 P fe80663143b84f279b195cc24f47184a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd15815082e48c996faf4462d0f18c8 in term 0.
I20260619 14:41:21.090747 19334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f4cffc43854a3991a47591cf873036" candidate_uuid: "ebd15815082e48c996faf4462d0f18c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3871466d01243979ed8454e10569ebc" is_pre_election: true
I20260619 14:41:21.091024 19334 raft_consensus.cc:2454] T e0f4cffc43854a3991a47591cf873036 P c3871466d01243979ed8454e10569ebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebd15815082e48c996faf4462d0f18c8 in term 0.
I20260619 14:41:21.091144 19399 leader_election.cc:302] T e0f4cffc43854a3991a47591cf873036 P ebd15815082e48c996faf4462d0f18c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ebd15815082e48c996faf4462d0f18c8, fe80663143b84f279b195cc24f47184a; no voters:
I20260619 14:41:21.091426 19536 raft_consensus.cc:2781] T e0f4cffc43854a3991a47591cf873036 P ebd15815082e48c996faf4462d0f18c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:41:21.091514 19536 raft_consensus.cc:482] T e0f4cffc43854a3991a47591cf873036 P ebd15815082e48c996faf4462d0f18c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:41:21.091563 19536 raft_consensus.cc:3037] T e0f4cffc43854a3991a47591cf873036 P ebd15815082e48c996faf4462d0f18c8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:41:21.092518 19536 raft_consensus.cc:504] T e0f4cffc43854a3991a47591cf873036 P ebd15815082e48c996faf4462d0f18c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd15815082e48c996faf4462d0f18c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34151 } } peers { permanent_uuid: "fe80663143b84f279b195cc24f47184a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44453 } } peers { permanent_uuid: "c3871466d01243979ed8454e10569ebc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41581 } }
I20260619 14:41:21.092846 19536 leader_election.cc:288] T e0f4cffc43854a3991a47591cf873036 P ebd15815082e48c996faf4462d0f18c8 [CANDIDATE]: Term 1 election: Requested vote from peers fe80663143b84f279b195cc24f47184a (127.0.98.1:44453), c3871466d01243979ed8454e10569ebc (127.0.98.2:41581)
I20260619 14:41:21.093168 19203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f4cffc43854a3991a47591cf873036" candidate_uuid: "ebd15815082e48c996faf4462d0f18c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe80663143b84f279b195cc24f47184a"
I20260619 14:41:21.093250 19334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f4cffc43854a3991a47591cf873036" candidate_uuid: "ebd15815082e48c996faf4462d0f18c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3871466d01243979ed8454e10569ebc"
I20260619 14:41:21.093304 19203 raft_consensus.cc:3037] T e0f4cffc43854a3991a47591cf873036 P fe80663143b84f279b195cc24f47184a [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:41:21.093359 19334 raft_consensus.cc:3037] T e0f4cffc43854a3991a47591cf873036 P c3871466d01243979ed8454e10569ebc [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:41:21.094518 19334 raft_consensus.cc:2454] T e0f4cffc43854a3991a47591cf873036 P c3871466d01243979ed8454e10569ebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebd15815082e48c996faf4462d0f18c8 in term 1.
I20260619 14:41:21.094518 19203 raft_consensus.cc:2454] T e0f4cffc43854a3991a47591cf873036 P fe80663143b84f279b195cc24f47184a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebd15815082e48c996faf4462d0f18c8 in term 1.
I20260619 14:41:21.094794 19399 leader_election.cc:302] T e0f4cffc43854a3991a47591cf873036 P ebd15815082e48c996faf4462d0f18c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ebd15815082e48c996faf4462d0f18c8, fe80663143b84f279b195cc24f47184a; no voters:
I20260619 14:41:21.094937 19536 raft_consensus.cc:2781] T e0f4cffc43854a3991a47591cf873036 P ebd15815082e48c996faf4462d0f18c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:41:21.095150 19536 raft_consensus.cc:686] T e0f4cffc43854a3991a47591cf873036 P ebd15815082e48c996faf4462d0f18c8 [term 1 LEADER]: Becoming Leader. State: Replica: ebd15815082e48c996faf4462d0f18c8, State: Running, Role: LEADER
I20260619 14:41:21.095470 19536 consensus_queue.cc:237] T e0f4cffc43854a3991a47591cf873036 P ebd15815082e48c996faf4462d0f18c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd15815082e48c996faf4462d0f18c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34151 } } peers { permanent_uuid: "fe80663143b84f279b195cc24f47184a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44453 } } peers { permanent_uuid: "c3871466d01243979ed8454e10569ebc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41581 } }
I20260619 14:41:21.098531 19059 catalog_manager.cc:5495] T e0f4cffc43854a3991a47591cf873036 P ebd15815082e48c996faf4462d0f18c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ebd15815082e48c996faf4462d0f18c8 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "ebd15815082e48c996faf4462d0f18c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd15815082e48c996faf4462d0f18c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe80663143b84f279b195cc24f47184a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3871466d01243979ed8454e10569ebc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41581 } health_report { overall_health: UNKNOWN } } }
I20260619 14:41:21.111963 19524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:21.108570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46932 (local address 127.0.98.3:34151)
0619 14:41:21.108679 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:21.108683 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:21.108695 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:41:21.108859 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:21.108861 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:21.108914 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:21.108998 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:41:21.109235 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:21.109734 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:21.110580 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:21.110795 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:21.110883 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:21.111121 (+ 238us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:41:21.111512 (+ 391us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:41:21.111614 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:21.111799 (+ 185us) server_negotiation.cc:299] Negotiation successful
0619 14:41:21.111840 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 14:41:21.117873 19334 raft_consensus.cc:1261] T e0f4cffc43854a3991a47591cf873036 P c3871466d01243979ed8454e10569ebc [term 1 FOLLOWER]: Refusing update from remote peer ebd15815082e48c996faf4462d0f18c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:41:21.117868 19203 raft_consensus.cc:1261] T e0f4cffc43854a3991a47591cf873036 P fe80663143b84f279b195cc24f47184a [term 1 FOLLOWER]: Refusing update from remote peer ebd15815082e48c996faf4462d0f18c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:41:21.118692 19538 consensus_queue.cc:1035] T e0f4cffc43854a3991a47591cf873036 P ebd15815082e48c996faf4462d0f18c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe80663143b84f279b195cc24f47184a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:41:21.118891 19536 consensus_queue.cc:1035] T e0f4cffc43854a3991a47591cf873036 P ebd15815082e48c996faf4462d0f18c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3871466d01243979ed8454e10569ebc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:41:21.126364 19541 mvcc.cc:205] Tried to move back new op lower bound from 7298580812250923008 to 7298580812169269248. Current Snapshot: MvccSnapshot[applied={T|T < 7298580812250923008}]
I20260619 14:41:21.128466 19543 mvcc.cc:205] Tried to move back new op lower bound from 7298580812250923008 to 7298580812169269248. Current Snapshot: MvccSnapshot[applied={T|T < 7298580812250923008}]
I20260619 14:41:21.135924 19059 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35096:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:41:21.136138 19059 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35096:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:41:21.139003 19059 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3542f6461b4541d8ba6092203af44c29: Sending DeleteTablet for 3 replicas of tablet e0f4cffc43854a3991a47591cf873036
I20260619 14:41:21.140012 19314 tablet_service.cc:1514] Processing DeleteTablet for tablet e0f4cffc43854a3991a47591cf873036 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:41:21 UTC) from {username='slave'} at 127.0.0.1:48302
I20260619 14:41:21.140118 19445 tablet_service.cc:1514] Processing DeleteTablet for tablet e0f4cffc43854a3991a47591cf873036 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:41:21 UTC) from {username='slave'} at 127.0.0.1:46926
I20260619 14:41:21.140100 19183 tablet_service.cc:1514] Processing DeleteTablet for tablet e0f4cffc43854a3991a47591cf873036 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:41:21 UTC) from {username='slave'} at 127.0.0.1:52262
I20260619 14:41:21.140626 19550 tablet_replica.cc:330] T e0f4cffc43854a3991a47591cf873036 P c3871466d01243979ed8454e10569ebc: stopping tablet replica
I20260619 14:41:21.140939 19552 tablet_replica.cc:330] T e0f4cffc43854a3991a47591cf873036 P fe80663143b84f279b195cc24f47184a: stopping tablet replica
I20260619 14:41:21.141013 19550 raft_consensus.cc:2229] T e0f4cffc43854a3991a47591cf873036 P c3871466d01243979ed8454e10569ebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:41:21.141044 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 19121
I20260619 14:41:21.140626 19551 tablet_replica.cc:330] T e0f4cffc43854a3991a47591cf873036 P ebd15815082e48c996faf4462d0f18c8: stopping tablet replica
I20260619 14:41:21.141698 19551 raft_consensus.cc:2229] T e0f4cffc43854a3991a47591cf873036 P ebd15815082e48c996faf4462d0f18c8 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:41:21.141942 19550 raft_consensus.cc:2258] T e0f4cffc43854a3991a47591cf873036 P c3871466d01243979ed8454e10569ebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:41:21.142256 19551 raft_consensus.cc:2258] T e0f4cffc43854a3991a47591cf873036 P ebd15815082e48c996faf4462d0f18c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:41:21.143667 19550 ts_tablet_manager.cc:1905] T e0f4cffc43854a3991a47591cf873036 P c3871466d01243979ed8454e10569ebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:41:21.143925 19551 ts_tablet_manager.cc:1905] T e0f4cffc43854a3991a47591cf873036 P ebd15815082e48c996faf4462d0f18c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:41:21.146340 19550 ts_tablet_manager.cc:1918] T e0f4cffc43854a3991a47591cf873036 P c3871466d01243979ed8454e10569ebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:41:21.146432 19550 log.cc:1196] T e0f4cffc43854a3991a47591cf873036 P c3871466d01243979ed8454e10569ebc: Deleting WAL directory at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/ts-1/wal/wals/e0f4cffc43854a3991a47591cf873036
I20260619 14:41:21.146665 19551 ts_tablet_manager.cc:1918] T e0f4cffc43854a3991a47591cf873036 P ebd15815082e48c996faf4462d0f18c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:41:21.146710 19550 ts_tablet_manager.cc:1939] T e0f4cffc43854a3991a47591cf873036 P c3871466d01243979ed8454e10569ebc: Deleting consensus metadata
I20260619 14:41:21.146839 19551 log.cc:1196] T e0f4cffc43854a3991a47591cf873036 P ebd15815082e48c996faf4462d0f18c8: Deleting WAL directory at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781879837648036-392-0/minicluster-data/ts-2/wal/wals/e0f4cffc43854a3991a47591cf873036
I20260619 14:41:21.147178 19551 ts_tablet_manager.cc:1939] T e0f4cffc43854a3991a47591cf873036 P ebd15815082e48c996faf4462d0f18c8: Deleting consensus metadata
I20260619 14:41:21.148383 19047 catalog_manager.cc:4841] TS ebd15815082e48c996faf4462d0f18c8 (127.0.98.3:34151): tablet e0f4cffc43854a3991a47591cf873036 (table test-table [id=1a541c4cab114207886122955d205f6a]) successfully deleted
W20260619 14:41:21.149983 19046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44453 (error 108)
I20260619 14:41:21.150375 19046 catalog_manager.cc:4841] TS c3871466d01243979ed8454e10569ebc (127.0.98.2:41581): tablet e0f4cffc43854a3991a47591cf873036 (table test-table [id=1a541c4cab114207886122955d205f6a]) successfully deleted
I20260619 14:41:21.150852 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 19252
I20260619 14:41:21.150902 19523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:21.150629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44453 (local address 127.0.0.1:52268)
0619 14:41:21.150749 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:21.150817 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 14:41:21.151511 19046 catalog_manager.cc:4584] TS fe80663143b84f279b195cc24f47184a (127.0.98.1:44453): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e0f4cffc43854a3991a47591cf873036: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44453: connect: Connection refused (error 111)
I20260619 14:41:21.155390 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 19383
I20260619 14:41:21.160486 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 19030
2026-06-19T14:41:21Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1188 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:41:21 dist-test-slave-32hj krb5kdc[19556](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:41:21 dist-test-slave-32hj krb5kdc[19556](info): set up 2 sockets
Jun 19 14:41:21 dist-test-slave-32hj krb5kdc[19556](info): commencing operation
krb5kdc: starting...
W20260619 14:41:22.104462 392 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/krb5kdc/test-user.keytab.
Jun 19 14:41:22 dist-test-slave-32hj krb5kdc[19556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:41:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:41:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:41:22.225378 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42763
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42763
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:41:22.332624 19569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:41:22.332913 19569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:41:22.332957 19569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:41:22.336882 19569 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:41:22.336942 19569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:41:22.336961 19569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:41:22.336975 19569 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:41:22.336989 19569 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:41:22.342294 19569 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42763
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:41:22.343376 19569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:41:22.344411 19569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:41:22.352705 19575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:22.352699 19577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:22.352865 19569 server_base.cc:1034] running on GCE node
W20260619 14:41:22.352699 19574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:22.353626 19569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:41:22.354735 19569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:41:22.355934 19569 hybrid_clock.cc:648] HybridClock initialized: now 1781880082355909 us; error 35 us; skew 500 ppm
Jun 19 14:41:22 dist-test-slave-32hj krb5kdc[19556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:41:22.363029 19569 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:41:22.363909 19569 webserver.cc:460] Webserver started at http://127.0.98.62:39787/ using document root <none> and password file <none>
I20260619 14:41:22.364440 19569 fs_manager.cc:359] Metadata directory not provided
I20260619 14:41:22.364498 19569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:41:22.364666 19569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:41:22.366686 19569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/master-0/data/instance:
uuid: "4e540ae00d824f739976958505f50fcf"
format_stamp: "Formatted at 2026-06-19 14:41:22 on dist-test-slave-32hj"
server_key: "ca14739aed38aa4b7abe8a6c4335c66f"
server_key_iv: "0b751868432c84e445c244bffb5064fc"
server_key_version: "encryptionkey@0"
I20260619 14:41:22.367180 19569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/master-0/wal/instance:
uuid: "4e540ae00d824f739976958505f50fcf"
format_stamp: "Formatted at 2026-06-19 14:41:22 on dist-test-slave-32hj"
server_key: "ca14739aed38aa4b7abe8a6c4335c66f"
server_key_iv: "0b751868432c84e445c244bffb5064fc"
server_key_version: "encryptionkey@0"
I20260619 14:41:22.370857 19569 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260619 14:41:22.373144 19584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:22.374420 19569 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:41:22.374519 19569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/master-0/wal
uuid: "4e540ae00d824f739976958505f50fcf"
format_stamp: "Formatted at 2026-06-19 14:41:22 on dist-test-slave-32hj"
server_key: "ca14739aed38aa4b7abe8a6c4335c66f"
server_key_iv: "0b751868432c84e445c244bffb5064fc"
server_key_version: "encryptionkey@0"
I20260619 14:41:22.374616 19569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:41:22.382944 19569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:41:22.385689 19569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:41:22.385828 19569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:41:22.392242 19569 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42763
I20260619 14:41:22.392283 19636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42763 every 8 connection(s)
I20260619 14:41:22.393191 19569 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:41:22.395855 19637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:41:22.401189 19637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e540ae00d824f739976958505f50fcf: Bootstrap starting.
I20260619 14:41:22.401262 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 19569
I20260619 14:41:22.401369 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/master-0/wal/instance
I20260619 14:41:22.401630 392 external_mini_cluster.cc:1428] Setting key e03e59b0c71280615094a046691fec45
I20260619 14:41:22.403621 19637 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e540ae00d824f739976958505f50fcf: Neither blocks nor log segments found. Creating new log.
I20260619 14:41:22.404330 19637 log.cc:824] T 00000000000000000000000000000000 P 4e540ae00d824f739976958505f50fcf: Log is configured to *not* fsync() on all Append() calls
I20260619 14:41:22.406122 19637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e540ae00d824f739976958505f50fcf: No bootstrap required, opened a new log
I20260619 14:41:22.409638 19637 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e540ae00d824f739976958505f50fcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e540ae00d824f739976958505f50fcf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42763 } }
I20260619 14:41:22.409940 19637 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e540ae00d824f739976958505f50fcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:41:22.410008 19637 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e540ae00d824f739976958505f50fcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e540ae00d824f739976958505f50fcf, State: Initialized, Role: FOLLOWER
Jun 19 14:41:22 dist-test-slave-32hj krb5kdc[19556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:41:22.410477 19637 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e540ae00d824f739976958505f50fcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e540ae00d824f739976958505f50fcf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42763 } }
I20260619 14:41:22.410612 19637 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e540ae00d824f739976958505f50fcf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:41:22.410676 19637 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e540ae00d824f739976958505f50fcf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:41:22.410771 19637 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e540ae00d824f739976958505f50fcf [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:41:22.411818 19637 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e540ae00d824f739976958505f50fcf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e540ae00d824f739976958505f50fcf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42763 } }
I20260619 14:41:22.412142 19637 leader_election.cc:302] T 00000000000000000000000000000000 P 4e540ae00d824f739976958505f50fcf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e540ae00d824f739976958505f50fcf; no voters:
I20260619 14:41:22.412432 19637 leader_election.cc:288] T 00000000000000000000000000000000 P 4e540ae00d824f739976958505f50fcf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:41:22.412546 19642 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e540ae00d824f739976958505f50fcf [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:41:22.412796 19642 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e540ae00d824f739976958505f50fcf [term 1 LEADER]: Becoming Leader. State: Replica: 4e540ae00d824f739976958505f50fcf, State: Running, Role: LEADER
I20260619 14:41:22.413103 19642 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e540ae00d824f739976958505f50fcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e540ae00d824f739976958505f50fcf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42763 } }
I20260619 14:41:22.413756 19637 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e540ae00d824f739976958505f50fcf [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:41:22.414326 19642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e540ae00d824f739976958505f50fcf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e540ae00d824f739976958505f50fcf. Latest consensus state: current_term: 1 leader_uuid: "4e540ae00d824f739976958505f50fcf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e540ae00d824f739976958505f50fcf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42763 } } }
I20260619 14:41:22.414436 19642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e540ae00d824f739976958505f50fcf [sys.catalog]: This master's current role is: LEADER
I20260619 14:41:22.414747 19643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e540ae00d824f739976958505f50fcf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e540ae00d824f739976958505f50fcf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e540ae00d824f739976958505f50fcf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42763 } } }
I20260619 14:41:22.414842 19643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e540ae00d824f739976958505f50fcf [sys.catalog]: This master's current role is: LEADER
I20260619 14:41:22.414973 19648 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:41:22.416285 19640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:22.402924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53546 (local address 127.0.98.62:42763)
0619 14:41:22.403350 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:22.403365 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:22.403400 (+ 35us) server_negotiation.cc:407] Connection header received
0619 14:41:22.404042 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:22.404061 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:22.404404 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:22.404774 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:22.405534 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:22.406552 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:22.407392 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:22.407898 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:22.408053 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:22.410669 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:22.410675 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:22.410693 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:41:22.410726 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:22.412533 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:22.413066 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:22.413072 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:22.413078 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:22.413221 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:22.413515 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:22.413519 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:22.413520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:22.414020 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:22.414242 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:22.414934 (+ 692us) server_negotiation.cc:299] Negotiation successful
0619 14:41:22.415246 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":127,"threads_started":1}
I20260619 14:41:22.418074 19648 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:41:22.423293 19648 catalog_manager.cc:1347] Generated new cluster ID: a39a9c2901f649289ab2e3e6ef5c3918
I20260619 14:41:22.423357 19648 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:41:22.433789 19648 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:41:22.434966 19648 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:41:22.445470 19648 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e540ae00d824f739976958505f50fcf: Generated new TSK 0
I20260619 14:41:22.446156 19648 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 14:41:22.475618 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42763
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:41:22.589218 19663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:41:22.589449 19663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:41:22.589490 19663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:41:22.593441 19663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:41:22.593489 19663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:41:22.593554 19663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:41:22.598564 19663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:41:22.599530 19663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:41:22.600476 19663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:41:22.609611 19663 server_base.cc:1034] running on GCE node
W20260619 14:41:22.609562 19671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:22.609573 19668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:22.609562 19669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:22.610132 19663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:41:22.610740 19663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:41:22.611894 19663 hybrid_clock.cc:648] HybridClock initialized: now 1781880082611867 us; error 39 us; skew 500 ppm
Jun 19 14:41:22 dist-test-slave-32hj krb5kdc[19556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:41:22.614533 19663 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:41:22.615429 19663 webserver.cc:460] Webserver started at http://127.0.98.1:38603/ using document root <none> and password file <none>
I20260619 14:41:22.615969 19663 fs_manager.cc:359] Metadata directory not provided
I20260619 14:41:22.616031 19663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:41:22.616225 19663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:41:22.618182 19663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/ts-0/data/instance:
uuid: "a0e8a5d3d63f4f7c8ae78c1eaf6caa27"
format_stamp: "Formatted at 2026-06-19 14:41:22 on dist-test-slave-32hj"
server_key: "6400fff6d739efa27dbd763c9725a6b4"
server_key_iv: "df9a9843a329c4dd903a9094390c41d7"
server_key_version: "encryptionkey@0"
I20260619 14:41:22.618695 19663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a0e8a5d3d63f4f7c8ae78c1eaf6caa27"
format_stamp: "Formatted at 2026-06-19 14:41:22 on dist-test-slave-32hj"
server_key: "6400fff6d739efa27dbd763c9725a6b4"
server_key_iv: "df9a9843a329c4dd903a9094390c41d7"
server_key_version: "encryptionkey@0"
I20260619 14:41:22.622370 19663 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:41:22.624634 19678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:22.625762 19663 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:41:22.625867 19663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/ts-0/wal
uuid: "a0e8a5d3d63f4f7c8ae78c1eaf6caa27"
format_stamp: "Formatted at 2026-06-19 14:41:22 on dist-test-slave-32hj"
server_key: "6400fff6d739efa27dbd763c9725a6b4"
server_key_iv: "df9a9843a329c4dd903a9094390c41d7"
server_key_version: "encryptionkey@0"
I20260619 14:41:22.625942 19663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:41:22.636319 19663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:41:22.639117 19663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:41:22.639276 19663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:41:22.639791 19663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:41:22.640530 19663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:41:22.640573 19663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:22.640614 19663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:41:22.640640 19663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:22.648263 19663 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37261
I20260619 14:41:22.648309 19791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37261 every 8 connection(s)
I20260619 14:41:22.649106 19663 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:41:22.652073 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 19663
I20260619 14:41:22.652204 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:41:22.652500 392 external_mini_cluster.cc:1428] Setting key 4e2ad5dcfd13c58857975c16bd0f8c9e
Jun 19 14:41:22 dist-test-slave-32hj krb5kdc[19556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 14:41:22.662319 19640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:22.650690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47293 (local address 127.0.98.62:42763)
0619 14:41:22.650838 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:22.650843 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:22.651634 (+ 791us) server_negotiation.cc:407] Connection header received
0619 14:41:22.653242 (+ 1608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:22.653244 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:22.653286 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:22.653351 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:22.654748 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:22.655284 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:22.656415 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:22.656645 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:22.656748 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:22.659103 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:22.659106 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:22.659107 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:41:22.659130 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:22.660444 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:22.661015 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:22.661019 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:22.661021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:22.661094 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:22.661406 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:22.661407 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:22.661409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:22.661560 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:22.661627 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:22.662104 (+ 477us) server_negotiation.cc:299] Negotiation successful
0619 14:41:22.662200 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 14:41:22.663187 19794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:22.651069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42763 (local address 127.0.98.1:47293)
0619 14:41:22.651496 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:22.651537 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:22.652912 (+ 1375us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:22.653549 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:22.653557 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:22.653955 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:41:22.654592 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:22.654612 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:22.655405 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:22.655408 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:22.656276 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:22.656310 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:22.656803 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:22.656807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:22.656963 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:22.657476 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:41:22.657492 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:41:22.658942 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:41:22.660558 (+ 1616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:41:22.660563 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:41:22.660578 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:41:22.660922 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:41:22.661179 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:41:22.661182 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:41:22.661184 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:41:22.661331 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:41:22.661629 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:41:22.661634 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:41:22.661911 (+ 277us) client_negotiation.cc:769] Sending connection context
0619 14:41:22.662053 (+ 142us) client_negotiation.cc:240] Negotiation successful
0619 14:41:22.662287 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":111,"threads_started":1}
I20260619 14:41:22.664302 19792 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42763
I20260619 14:41:22.664551 19792 heartbeater.cc:464] Registering TS with master...
I20260619 14:41:22.665148 19792 heartbeater.cc:511] Master 127.0.98.62:42763 requested a full tablet report, sending...
I20260619 14:41:22.666567 19601 ts_manager.cc:194] Registered new tserver with Master: a0e8a5d3d63f4f7c8ae78c1eaf6caa27 (127.0.98.1:37261)
I20260619 14:41:22.667646 19601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47293
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 14:41:22.675483 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42763
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:41:22.778854 19797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:41:22.779093 19797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:41:22.779134 19797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:41:22.783080 19797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:41:22.783136 19797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:41:22.783191 19797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:41:22.788182 19797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:41:22.789152 19797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:41:22.790109 19797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:41:22.799265 19797 server_base.cc:1034] running on GCE node
W20260619 14:41:22.799235 19802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:22.799249 19803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:22.799461 19805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:22.799858 19797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:41:22.800467 19797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:41:22.801615 19797 hybrid_clock.cc:648] HybridClock initialized: now 1781880082801575 us; error 52 us; skew 500 ppm
Jun 19 14:41:22 dist-test-slave-32hj krb5kdc[19556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:41:22.804232 19797 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:41:22.805135 19797 webserver.cc:460] Webserver started at http://127.0.98.2:33569/ using document root <none> and password file <none>
I20260619 14:41:22.805667 19797 fs_manager.cc:359] Metadata directory not provided
I20260619 14:41:22.805729 19797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:41:22.805927 19797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:41:22.807843 19797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/ts-1/data/instance:
uuid: "39d1bf2b1e254c3987a4670cc0ea3d08"
format_stamp: "Formatted at 2026-06-19 14:41:22 on dist-test-slave-32hj"
server_key: "6be2837805b4bb1114e86a474bda7c0d"
server_key_iv: "3a0af9c43cf8c55841b5deeb7db3b01e"
server_key_version: "encryptionkey@0"
I20260619 14:41:22.808348 19797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/ts-1/wal/instance:
uuid: "39d1bf2b1e254c3987a4670cc0ea3d08"
format_stamp: "Formatted at 2026-06-19 14:41:22 on dist-test-slave-32hj"
server_key: "6be2837805b4bb1114e86a474bda7c0d"
server_key_iv: "3a0af9c43cf8c55841b5deeb7db3b01e"
server_key_version: "encryptionkey@0"
I20260619 14:41:22.811764 19797 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 14:41:22.814110 19812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:22.815392 19797 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:41:22.815498 19797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/ts-1/wal
uuid: "39d1bf2b1e254c3987a4670cc0ea3d08"
format_stamp: "Formatted at 2026-06-19 14:41:22 on dist-test-slave-32hj"
server_key: "6be2837805b4bb1114e86a474bda7c0d"
server_key_iv: "3a0af9c43cf8c55841b5deeb7db3b01e"
server_key_version: "encryptionkey@0"
I20260619 14:41:22.815582 19797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:41:22.847687 19797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:41:22.850896 19797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:41:22.851066 19797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:41:22.851639 19797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:41:22.852447 19797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:41:22.852491 19797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:22.852542 19797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:41:22.852571 19797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:22.860743 19797 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37033
I20260619 14:41:22.860780 19925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37033 every 8 connection(s)
I20260619 14:41:22.861601 19797 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 14:41:22 dist-test-slave-32hj krb5kdc[19556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:41:22.870973 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 19797
I20260619 14:41:22.871109 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:41:22.871423 392 external_mini_cluster.cc:1428] Setting key 41c8a9522f9e913b3ec2406d61f05627
I20260619 14:41:22.874608 19640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:22.863301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44003 (local address 127.0.98.62:42763)
0619 14:41:22.863436 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:22.863441 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:22.864254 (+ 813us) server_negotiation.cc:407] Connection header received
0619 14:41:22.865494 (+ 1240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:22.865497 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:22.865542 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:22.865615 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:22.866709 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:22.867214 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:22.868285 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:22.868484 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:22.868552 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:22.870894 (+ 2342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:22.870899 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:22.870901 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:41:22.870931 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:22.872545 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:22.873206 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:22.873210 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:22.873212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:22.873289 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:22.873603 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:22.873605 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:22.873606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:22.873779 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:22.873859 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:22.874341 (+ 482us) server_negotiation.cc:299] Negotiation successful
0619 14:41:22.874455 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 14:41:22.875392 19928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:22.863682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42763 (local address 127.0.98.2:44003)
0619 14:41:22.864114 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:22.864156 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:22.865311 (+ 1155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:22.865763 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:22.865770 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:22.866104 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:41:22.866595 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:22.866608 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:22.867323 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:22.867325 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:22.868184 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:22.868192 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:22.868610 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:22.868613 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:22.868795 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:22.869317 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:41:22.869333 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:41:22.870722 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:41:22.872776 (+ 2054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:41:22.872782 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:41:22.872796 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:41:22.873111 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:41:22.873372 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:41:22.873375 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:41:22.873377 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:41:22.873510 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:41:22.873875 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:41:22.873882 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:41:22.874111 (+ 229us) client_negotiation.cc:769] Sending connection context
0619 14:41:22.874269 (+ 158us) client_negotiation.cc:240] Negotiation successful
0619 14:41:22.874574 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":109,"threads_started":1}
I20260619 14:41:22.876428 19926 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42763
I20260619 14:41:22.876737 19926 heartbeater.cc:464] Registering TS with master...
I20260619 14:41:22.877283 19926 heartbeater.cc:511] Master 127.0.98.62:42763 requested a full tablet report, sending...
I20260619 14:41:22.878310 19601 ts_manager.cc:194] Registered new tserver with Master: 39d1bf2b1e254c3987a4670cc0ea3d08 (127.0.98.2:37033)
I20260619 14:41:22.879076 19601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44003
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 14:41:22.893819 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42763
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:41:23.002046 19931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:41:23.002457 19931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:41:23.002533 19931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:41:23.007967 19931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:41:23.008049 19931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:41:23.008143 19931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:41:23.013690 19931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:41:23.014796 19931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:41:23.015858 19931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:41:23.025274 19939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:23.025225 19936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:23.025415 19931 server_base.cc:1034] running on GCE node
W20260619 14:41:23.025230 19937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:23.025967 19931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:41:23.026575 19931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:41:23.027756 19931 hybrid_clock.cc:648] HybridClock initialized: now 1781880083027737 us; error 88 us; skew 500 ppm
Jun 19 14:41:23 dist-test-slave-32hj krb5kdc[19556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:41:23.030650 19931 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:41:23.031495 19931 webserver.cc:460] Webserver started at http://127.0.98.3:40505/ using document root <none> and password file <none>
I20260619 14:41:23.032023 19931 fs_manager.cc:359] Metadata directory not provided
I20260619 14:41:23.032087 19931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:41:23.032307 19931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:41:23.034317 19931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/ts-2/data/instance:
uuid: "d896c150328d4d068bc22f2abdb68bc1"
format_stamp: "Formatted at 2026-06-19 14:41:23 on dist-test-slave-32hj"
server_key: "e35706501daaa790d8a5733654fedaba"
server_key_iv: "16c10db0d161047dd26eca451bdf568e"
server_key_version: "encryptionkey@0"
I20260619 14:41:23.034844 19931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d896c150328d4d068bc22f2abdb68bc1"
format_stamp: "Formatted at 2026-06-19 14:41:23 on dist-test-slave-32hj"
server_key: "e35706501daaa790d8a5733654fedaba"
server_key_iv: "16c10db0d161047dd26eca451bdf568e"
server_key_version: "encryptionkey@0"
I20260619 14:41:23.038676 19931 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260619 14:41:23.041074 19946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:23.042268 19931 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260619 14:41:23.042372 19931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/ts-2/wal
uuid: "d896c150328d4d068bc22f2abdb68bc1"
format_stamp: "Formatted at 2026-06-19 14:41:23 on dist-test-slave-32hj"
server_key: "e35706501daaa790d8a5733654fedaba"
server_key_iv: "16c10db0d161047dd26eca451bdf568e"
server_key_version: "encryptionkey@0"
I20260619 14:41:23.042471 19931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:41:23.060433 19931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:41:23.063589 19931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:41:23.063778 19931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:41:23.064395 19931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:41:23.065244 19931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:41:23.065291 19931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:23.065342 19931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:41:23.065369 19931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:23.073493 19931 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34521
I20260619 14:41:23.073527 20059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34521 every 8 connection(s)
I20260619 14:41:23.074330 19931 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:41:23.081771 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 19931
I20260619 14:41:23.081869 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:41:23.082113 392 external_mini_cluster.cc:1428] Setting key c97d2c7a37808dbaf28f591c7ed4f090
Jun 19 14:41:23 dist-test-slave-32hj krb5kdc[19556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:41:23.087383 19640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:23.075987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53825 (local address 127.0.98.62:42763)
0619 14:41:23.076119 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:23.076123 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:23.077013 (+ 890us) server_negotiation.cc:407] Connection header received
0619 14:41:23.078286 (+ 1273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:23.078289 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:23.078336 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:23.078411 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:23.079523 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:23.080071 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:23.081299 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:23.081556 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:23.081666 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:23.084306 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:23.084309 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:23.084311 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:41:23.084336 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:23.085513 (+ 1177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:23.085976 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:23.085978 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:23.085980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:23.086051 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:23.086357 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:23.086360 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:23.086362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:23.086531 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:23.086612 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:23.087093 (+ 481us) server_negotiation.cc:299] Negotiation successful
0619 14:41:23.087228 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 14:41:23.088151 20062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:23.076402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42763 (local address 127.0.98.3:53825)
0619 14:41:23.076872 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:23.076915 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:23.078028 (+ 1113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:23.078552 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:23.078559 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:23.078890 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:41:23.079368 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:23.079381 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:23.080198 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:23.080201 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:23.081115 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:23.081124 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:23.081736 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:23.081740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:23.081922 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:23.082506 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:41:23.082526 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:41:23.084153 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:41:23.085614 (+ 1461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:41:23.085620 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:41:23.085635 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:41:23.085882 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:41:23.086129 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:41:23.086132 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:41:23.086134 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:41:23.086274 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:41:23.086624 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:41:23.086631 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:41:23.086873 (+ 242us) client_negotiation.cc:769] Sending connection context
0619 14:41:23.087036 (+ 163us) client_negotiation.cc:240] Negotiation successful
0619 14:41:23.087352 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":117,"threads_started":1}
I20260619 14:41:23.089160 20060 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42763
I20260619 14:41:23.089392 20060 heartbeater.cc:464] Registering TS with master...
I20260619 14:41:23.089838 20060 heartbeater.cc:511] Master 127.0.98.62:42763 requested a full tablet report, sending...
I20260619 14:41:23.090802 19601 ts_manager.cc:194] Registered new tserver with Master: d896c150328d4d068bc22f2abdb68bc1 (127.0.98.3:34521)
I20260619 14:41:23.091342 19601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53825
I20260619 14:41:23.095402 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:41:23.103405 19640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:23.097124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53558 (local address 127.0.98.62:42763)
0619 14:41:23.097242 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:23.097246 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:23.097394 (+ 148us) server_negotiation.cc:407] Connection header received
0619 14:41:23.097478 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:23.097480 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:23.097513 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:23.097569 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:23.098127 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:23.098614 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:23.099355 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:23.099559 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:23.099629 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:23.100931 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:23.100934 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:23.100936 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:41:23.100957 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:23.102026 (+ 1069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:23.102399 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:23.102401 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:23.102403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:23.102467 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:23.102648 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:23.102650 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:23.102652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:23.102841 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:23.102935 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:23.103182 (+ 247us) server_negotiation.cc:299] Negotiation successful
0619 14:41:23.103269 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 14:41:23.105962 19601 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53558:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:41:23.108984 19601 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:41:23.122977 20074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:23.117063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34521 (local address 127.0.0.1:33942)
0619 14:41:23.118049 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:23.118100 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:23.118275 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:23.118981 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:23.118985 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:23.119013 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:41:23.119127 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:23.119133 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:23.120382 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:23.120385 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:23.121539 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:23.121548 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:23.122322 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:23.122325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:23.122543 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:23.122584 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 14:41:23.122690 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 14:41:23.122815 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":805,"mutex_wait_us":61,"thread_start_us":71,"threads_started":1}
I20260619 14:41:23.123001 20073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:23.117188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33942 (local address 127.0.98.3:34521)
0619 14:41:23.118478 (+ 1290us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:23.118488 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:23.118510 (+ 22us) server_negotiation.cc:407] Connection header received
0619 14:41:23.118586 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:23.118592 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:23.118822 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:23.118980 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:41:23.119326 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:23.120265 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:23.121660 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:23.122213 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:23.122309 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:23.122464 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:23.122729 (+ 265us) server_negotiation.cc:299] Negotiation successful
0619 14:41:23.122841 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1198,"thread_start_us":127,"threads_started":1}
I20260619 14:41:23.123970 20072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:23.117189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37261 (local address 127.0.0.1:33310)
0619 14:41:23.118228 (+ 1039us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:23.118242 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:23.119154 (+ 912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:23.119636 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:23.119639 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:23.119655 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:41:23.119744 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:23.119749 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:23.121062 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:23.121065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:23.122524 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:23.122535 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:23.123559 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:23.123562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:23.123676 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:23.123696 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 14:41:23.123773 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 14:41:23.123842 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":978,"mutex_wait_us":156,"thread_start_us":102,"threads_started":1}
I20260619 14:41:23.124610 20071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:23.117403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37033 (local address 127.0.0.1:37996)
0619 14:41:23.118474 (+ 1071us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:23.118486 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:23.118568 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:23.119288 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:23.119290 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:23.119308 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:41:23.119389 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:23.119395 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:23.121062 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:23.121065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:23.122525 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:23.122535 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:23.124287 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:23.124290 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:23.124379 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:23.124402 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 14:41:23.124463 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 14:41:23.124522 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1019,"mutex_wait_us":172,"thread_start_us":85,"threads_started":1}
I20260619 14:41:23.125064 20076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:23.117869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37996 (local address 127.0.98.2:37033)
0619 14:41:23.118290 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:23.118306 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:23.118559 (+ 253us) server_negotiation.cc:407] Connection header received
0619 14:41:23.118666 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:23.118670 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:23.118821 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:23.118980 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:41:23.119501 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:23.120914 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:23.122656 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:23.123434 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:23.123531 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:23.123815 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:23.124806 (+ 991us) server_negotiation.cc:299] Negotiation successful
0619 14:41:23.124939 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":99,"threads_started":1}
I20260619 14:41:23.126142 19994 tablet_service.cc:1467] Processing CreateTablet for tablet ced525acd241456598612a11e487849f (DEFAULT_TABLE table=test-table [id=9cdfbac9f0c44836af3db9891ad57173]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:41:23.127308 19994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ced525acd241456598612a11e487849f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:41:23.127770 19860 tablet_service.cc:1467] Processing CreateTablet for tablet ced525acd241456598612a11e487849f (DEFAULT_TABLE table=test-table [id=9cdfbac9f0c44836af3db9891ad57173]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:41:23.129007 19860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ced525acd241456598612a11e487849f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:41:23.132890 20075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:23.117851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33310 (local address 127.0.98.1:37261)
0619 14:41:23.118490 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:23.118504 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:23.118526 (+ 22us) server_negotiation.cc:407] Connection header received
0619 14:41:23.119310 (+ 784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:23.119317 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:23.119482 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:23.119637 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:41:23.119834 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:23.120937 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:23.122660 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:23.123454 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:23.123550 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:23.123681 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:23.123800 (+ 119us) server_negotiation.cc:299] Negotiation successful
0619 14:41:23.123881 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":95,"threads_started":1}
I20260619 14:41:23.134743 20077 tablet_bootstrap.cc:492] T ced525acd241456598612a11e487849f P d896c150328d4d068bc22f2abdb68bc1: Bootstrap starting.
I20260619 14:41:23.135341 19726 tablet_service.cc:1467] Processing CreateTablet for tablet ced525acd241456598612a11e487849f (DEFAULT_TABLE table=test-table [id=9cdfbac9f0c44836af3db9891ad57173]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:41:23.136297 20078 tablet_bootstrap.cc:492] T ced525acd241456598612a11e487849f P 39d1bf2b1e254c3987a4670cc0ea3d08: Bootstrap starting.
I20260619 14:41:23.136348 19726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ced525acd241456598612a11e487849f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:41:23.137002 20077 tablet_bootstrap.cc:654] T ced525acd241456598612a11e487849f P d896c150328d4d068bc22f2abdb68bc1: Neither blocks nor log segments found. Creating new log.
I20260619 14:41:23.138213 20077 log.cc:824] T ced525acd241456598612a11e487849f P d896c150328d4d068bc22f2abdb68bc1: Log is configured to *not* fsync() on all Append() calls
I20260619 14:41:23.138516 20078 tablet_bootstrap.cc:654] T ced525acd241456598612a11e487849f P 39d1bf2b1e254c3987a4670cc0ea3d08: Neither blocks nor log segments found. Creating new log.
I20260619 14:41:23.139233 20078 log.cc:824] T ced525acd241456598612a11e487849f P 39d1bf2b1e254c3987a4670cc0ea3d08: Log is configured to *not* fsync() on all Append() calls
I20260619 14:41:23.140259 20077 tablet_bootstrap.cc:492] T ced525acd241456598612a11e487849f P d896c150328d4d068bc22f2abdb68bc1: No bootstrap required, opened a new log
I20260619 14:41:23.140443 20077 ts_tablet_manager.cc:1397] T ced525acd241456598612a11e487849f P d896c150328d4d068bc22f2abdb68bc1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 14:41:23.140710 20078 tablet_bootstrap.cc:492] T ced525acd241456598612a11e487849f P 39d1bf2b1e254c3987a4670cc0ea3d08: No bootstrap required, opened a new log
I20260619 14:41:23.140908 20078 ts_tablet_manager.cc:1397] T ced525acd241456598612a11e487849f P 39d1bf2b1e254c3987a4670cc0ea3d08: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:41:23.143654 20081 tablet_bootstrap.cc:492] T ced525acd241456598612a11e487849f P a0e8a5d3d63f4f7c8ae78c1eaf6caa27: Bootstrap starting.
I20260619 14:41:23.143945 20078 raft_consensus.cc:348] T ced525acd241456598612a11e487849f P 39d1bf2b1e254c3987a4670cc0ea3d08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d896c150328d4d068bc22f2abdb68bc1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34521 } } peers { permanent_uuid: "a0e8a5d3d63f4f7c8ae78c1eaf6caa27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37261 } } peers { permanent_uuid: "39d1bf2b1e254c3987a4670cc0ea3d08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37033 } }
I20260619 14:41:23.144160 20078 raft_consensus.cc:374] T ced525acd241456598612a11e487849f P 39d1bf2b1e254c3987a4670cc0ea3d08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:41:23.144217 20078 raft_consensus.cc:729] T ced525acd241456598612a11e487849f P 39d1bf2b1e254c3987a4670cc0ea3d08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39d1bf2b1e254c3987a4670cc0ea3d08, State: Initialized, Role: FOLLOWER
I20260619 14:41:23.144105 20077 raft_consensus.cc:348] T ced525acd241456598612a11e487849f P d896c150328d4d068bc22f2abdb68bc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d896c150328d4d068bc22f2abdb68bc1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34521 } } peers { permanent_uuid: "a0e8a5d3d63f4f7c8ae78c1eaf6caa27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37261 } } peers { permanent_uuid: "39d1bf2b1e254c3987a4670cc0ea3d08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37033 } }
I20260619 14:41:23.144358 20077 raft_consensus.cc:374] T ced525acd241456598612a11e487849f P d896c150328d4d068bc22f2abdb68bc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:41:23.144416 20077 raft_consensus.cc:729] T ced525acd241456598612a11e487849f P d896c150328d4d068bc22f2abdb68bc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d896c150328d4d068bc22f2abdb68bc1, State: Initialized, Role: FOLLOWER
I20260619 14:41:23.144618 20078 consensus_queue.cc:260] T ced525acd241456598612a11e487849f P 39d1bf2b1e254c3987a4670cc0ea3d08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d896c150328d4d068bc22f2abdb68bc1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34521 } } peers { permanent_uuid: "a0e8a5d3d63f4f7c8ae78c1eaf6caa27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37261 } } peers { permanent_uuid: "39d1bf2b1e254c3987a4670cc0ea3d08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37033 } }
I20260619 14:41:23.144827 20077 consensus_queue.cc:260] T ced525acd241456598612a11e487849f P d896c150328d4d068bc22f2abdb68bc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d896c150328d4d068bc22f2abdb68bc1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34521 } } peers { permanent_uuid: "a0e8a5d3d63f4f7c8ae78c1eaf6caa27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37261 } } peers { permanent_uuid: "39d1bf2b1e254c3987a4670cc0ea3d08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37033 } }
I20260619 14:41:23.145268 19926 heartbeater.cc:503] Master 127.0.98.62:42763 was elected leader, sending a full tablet report...
I20260619 14:41:23.145682 20081 tablet_bootstrap.cc:654] T ced525acd241456598612a11e487849f P a0e8a5d3d63f4f7c8ae78c1eaf6caa27: Neither blocks nor log segments found. Creating new log.
I20260619 14:41:23.145716 20077 ts_tablet_manager.cc:1428] T ced525acd241456598612a11e487849f P d896c150328d4d068bc22f2abdb68bc1: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 14:41:23.146042 20060 heartbeater.cc:503] Master 127.0.98.62:42763 was elected leader, sending a full tablet report...
I20260619 14:41:23.146457 20078 ts_tablet_manager.cc:1428] T ced525acd241456598612a11e487849f P 39d1bf2b1e254c3987a4670cc0ea3d08: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260619 14:41:23.146785 20081 log.cc:824] T ced525acd241456598612a11e487849f P a0e8a5d3d63f4f7c8ae78c1eaf6caa27: Log is configured to *not* fsync() on all Append() calls
I20260619 14:41:23.148859 20081 tablet_bootstrap.cc:492] T ced525acd241456598612a11e487849f P a0e8a5d3d63f4f7c8ae78c1eaf6caa27: No bootstrap required, opened a new log
I20260619 14:41:23.149034 20081 ts_tablet_manager.cc:1397] T ced525acd241456598612a11e487849f P a0e8a5d3d63f4f7c8ae78c1eaf6caa27: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 14:41:23.151837 20081 raft_consensus.cc:348] T ced525acd241456598612a11e487849f P a0e8a5d3d63f4f7c8ae78c1eaf6caa27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d896c150328d4d068bc22f2abdb68bc1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34521 } } peers { permanent_uuid: "a0e8a5d3d63f4f7c8ae78c1eaf6caa27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37261 } } peers { permanent_uuid: "39d1bf2b1e254c3987a4670cc0ea3d08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37033 } }
I20260619 14:41:23.152021 20081 raft_consensus.cc:374] T ced525acd241456598612a11e487849f P a0e8a5d3d63f4f7c8ae78c1eaf6caa27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:41:23.152050 20081 raft_consensus.cc:729] T ced525acd241456598612a11e487849f P a0e8a5d3d63f4f7c8ae78c1eaf6caa27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0e8a5d3d63f4f7c8ae78c1eaf6caa27, State: Initialized, Role: FOLLOWER
I20260619 14:41:23.152400 20081 consensus_queue.cc:260] T ced525acd241456598612a11e487849f P a0e8a5d3d63f4f7c8ae78c1eaf6caa27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d896c150328d4d068bc22f2abdb68bc1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34521 } } peers { permanent_uuid: "a0e8a5d3d63f4f7c8ae78c1eaf6caa27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37261 } } peers { permanent_uuid: "39d1bf2b1e254c3987a4670cc0ea3d08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37033 } }
I20260619 14:41:23.152994 19792 heartbeater.cc:503] Master 127.0.98.62:42763 was elected leader, sending a full tablet report...
I20260619 14:41:23.153246 20081 ts_tablet_manager.cc:1428] T ced525acd241456598612a11e487849f P a0e8a5d3d63f4f7c8ae78c1eaf6caa27: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260619 14:41:23.249655 20083 raft_consensus.cc:482] T ced525acd241456598612a11e487849f P d896c150328d4d068bc22f2abdb68bc1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:41:23.249859 20083 raft_consensus.cc:504] T ced525acd241456598612a11e487849f P d896c150328d4d068bc22f2abdb68bc1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d896c150328d4d068bc22f2abdb68bc1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34521 } } peers { permanent_uuid: "a0e8a5d3d63f4f7c8ae78c1eaf6caa27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37261 } } peers { permanent_uuid: "39d1bf2b1e254c3987a4670cc0ea3d08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37033 } }
I20260619 14:41:23.251004 20083 leader_election.cc:288] T ced525acd241456598612a11e487849f P d896c150328d4d068bc22f2abdb68bc1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0e8a5d3d63f4f7c8ae78c1eaf6caa27 (127.0.98.1:37261), 39d1bf2b1e254c3987a4670cc0ea3d08 (127.0.98.2:37033)
I20260619 14:41:23.255872 20076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:23.251402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37591 (local address 127.0.98.2:37033)
0619 14:41:23.251525 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:23.251529 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:23.251718 (+ 189us) server_negotiation.cc:407] Connection header received
0619 14:41:23.251798 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:23.251801 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:23.251846 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:23.251909 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:41:23.252233 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:23.253054 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:23.254680 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:23.255289 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:23.255394 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:23.255457 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:23.255671 (+ 214us) server_negotiation.cc:299] Negotiation successful
0619 14:41:23.255741 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 14:41:23.255872 20075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:23.251356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50139 (local address 127.0.98.1:37261)
0619 14:41:23.251471 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:23.251475 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:23.251589 (+ 114us) server_negotiation.cc:407] Connection header received
0619 14:41:23.251702 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:23.251705 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:23.251758 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:23.251840 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:41:23.252237 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:23.253054 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:23.254682 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:23.255283 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:23.255392 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:23.255458 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:23.255681 (+ 223us) server_negotiation.cc:299] Negotiation successful
0619 14:41:23.255746 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 14:41:23.255905 20062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:23.251268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37261 (local address 127.0.98.3:50139)
0619 14:41:23.251451 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:23.251466 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:23.251575 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:23.251860 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:23.251862 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:23.251876 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:41:23.252054 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:23.252061 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:23.253180 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:23.253185 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:23.254532 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:23.254544 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:23.255401 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:23.255405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:23.255588 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:23.255605 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 14:41:23.255664 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 14:41:23.255755 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":104}
I20260619 14:41:23.255913 20086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:23.251295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37033 (local address 127.0.98.3:37591)
0619 14:41:23.251631 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:23.251643 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:23.251727 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:23.251926 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:23.251929 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:23.251940 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:41:23.252073 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:23.252078 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:23.253180 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:23.253185 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:23.254532 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:23.254544 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:23.255401 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:23.255405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:23.255536 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:23.255575 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 14:41:23.255651 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 14:41:23.255759 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"mutex_wait_us":161,"thread_start_us":142,"threads_started":1}
I20260619 14:41:23.256552 19746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ced525acd241456598612a11e487849f" candidate_uuid: "d896c150328d4d068bc22f2abdb68bc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0e8a5d3d63f4f7c8ae78c1eaf6caa27" is_pre_election: true
I20260619 14:41:23.256630 19880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ced525acd241456598612a11e487849f" candidate_uuid: "d896c150328d4d068bc22f2abdb68bc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39d1bf2b1e254c3987a4670cc0ea3d08" is_pre_election: true
I20260619 14:41:23.256865 19746 raft_consensus.cc:2454] T ced525acd241456598612a11e487849f P a0e8a5d3d63f4f7c8ae78c1eaf6caa27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d896c150328d4d068bc22f2abdb68bc1 in term 0.
I20260619 14:41:23.256920 19880 raft_consensus.cc:2454] T ced525acd241456598612a11e487849f P 39d1bf2b1e254c3987a4670cc0ea3d08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d896c150328d4d068bc22f2abdb68bc1 in term 0.
I20260619 14:41:23.257341 19948 leader_election.cc:302] T ced525acd241456598612a11e487849f P d896c150328d4d068bc22f2abdb68bc1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39d1bf2b1e254c3987a4670cc0ea3d08, d896c150328d4d068bc22f2abdb68bc1; no voters:
I20260619 14:41:23.257583 20083 raft_consensus.cc:2781] T ced525acd241456598612a11e487849f P d896c150328d4d068bc22f2abdb68bc1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:41:23.257668 20083 raft_consensus.cc:482] T ced525acd241456598612a11e487849f P d896c150328d4d068bc22f2abdb68bc1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:41:23.257716 20083 raft_consensus.cc:3037] T ced525acd241456598612a11e487849f P d896c150328d4d068bc22f2abdb68bc1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:41:23.258672 20083 raft_consensus.cc:504] T ced525acd241456598612a11e487849f P d896c150328d4d068bc22f2abdb68bc1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d896c150328d4d068bc22f2abdb68bc1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34521 } } peers { permanent_uuid: "a0e8a5d3d63f4f7c8ae78c1eaf6caa27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37261 } } peers { permanent_uuid: "39d1bf2b1e254c3987a4670cc0ea3d08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37033 } }
I20260619 14:41:23.258958 20083 leader_election.cc:288] T ced525acd241456598612a11e487849f P d896c150328d4d068bc22f2abdb68bc1 [CANDIDATE]: Term 1 election: Requested vote from peers a0e8a5d3d63f4f7c8ae78c1eaf6caa27 (127.0.98.1:37261), 39d1bf2b1e254c3987a4670cc0ea3d08 (127.0.98.2:37033)
I20260619 14:41:23.259289 19746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ced525acd241456598612a11e487849f" candidate_uuid: "d896c150328d4d068bc22f2abdb68bc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0e8a5d3d63f4f7c8ae78c1eaf6caa27"
I20260619 14:41:23.259289 19880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ced525acd241456598612a11e487849f" candidate_uuid: "d896c150328d4d068bc22f2abdb68bc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39d1bf2b1e254c3987a4670cc0ea3d08"
I20260619 14:41:23.259409 19880 raft_consensus.cc:3037] T ced525acd241456598612a11e487849f P 39d1bf2b1e254c3987a4670cc0ea3d08 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:41:23.259409 19746 raft_consensus.cc:3037] T ced525acd241456598612a11e487849f P a0e8a5d3d63f4f7c8ae78c1eaf6caa27 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:41:23.260558 19880 raft_consensus.cc:2454] T ced525acd241456598612a11e487849f P 39d1bf2b1e254c3987a4670cc0ea3d08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d896c150328d4d068bc22f2abdb68bc1 in term 1.
I20260619 14:41:23.260558 19746 raft_consensus.cc:2454] T ced525acd241456598612a11e487849f P a0e8a5d3d63f4f7c8ae78c1eaf6caa27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d896c150328d4d068bc22f2abdb68bc1 in term 1.
I20260619 14:41:23.260864 19950 leader_election.cc:302] T ced525acd241456598612a11e487849f P d896c150328d4d068bc22f2abdb68bc1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 39d1bf2b1e254c3987a4670cc0ea3d08, a0e8a5d3d63f4f7c8ae78c1eaf6caa27, d896c150328d4d068bc22f2abdb68bc1; no voters:
I20260619 14:41:23.261025 20083 raft_consensus.cc:2781] T ced525acd241456598612a11e487849f P d896c150328d4d068bc22f2abdb68bc1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:41:23.261255 20083 raft_consensus.cc:686] T ced525acd241456598612a11e487849f P d896c150328d4d068bc22f2abdb68bc1 [term 1 LEADER]: Becoming Leader. State: Replica: d896c150328d4d068bc22f2abdb68bc1, State: Running, Role: LEADER
I20260619 14:41:23.261572 20083 consensus_queue.cc:237] T ced525acd241456598612a11e487849f P d896c150328d4d068bc22f2abdb68bc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d896c150328d4d068bc22f2abdb68bc1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34521 } } peers { permanent_uuid: "a0e8a5d3d63f4f7c8ae78c1eaf6caa27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37261 } } peers { permanent_uuid: "39d1bf2b1e254c3987a4670cc0ea3d08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37033 } }
I20260619 14:41:23.264417 19601 catalog_manager.cc:5495] T ced525acd241456598612a11e487849f P d896c150328d4d068bc22f2abdb68bc1 reported cstate change: term changed from 0 to 1, leader changed from <none> to d896c150328d4d068bc22f2abdb68bc1 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "d896c150328d4d068bc22f2abdb68bc1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d896c150328d4d068bc22f2abdb68bc1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0e8a5d3d63f4f7c8ae78c1eaf6caa27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39d1bf2b1e254c3987a4670cc0ea3d08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37033 } health_report { overall_health: UNKNOWN } } }
I20260619 14:41:23.278252 20073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:23.274850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33958 (local address 127.0.98.3:34521)
0619 14:41:23.274966 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:23.274970 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:23.274983 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:41:23.275125 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:23.275128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:23.275183 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:23.275272 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:41:23.275507 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:23.275998 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:23.276850 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:23.277057 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:23.277142 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:23.277418 (+ 276us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:41:23.277880 (+ 462us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:41:23.277983 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:23.278041 (+ 58us) server_negotiation.cc:299] Negotiation successful
0619 14:41:23.278100 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 14:41:23.283962 19880 raft_consensus.cc:1261] T ced525acd241456598612a11e487849f P 39d1bf2b1e254c3987a4670cc0ea3d08 [term 1 FOLLOWER]: Refusing update from remote peer d896c150328d4d068bc22f2abdb68bc1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:41:23.283962 19746 raft_consensus.cc:1261] T ced525acd241456598612a11e487849f P a0e8a5d3d63f4f7c8ae78c1eaf6caa27 [term 1 FOLLOWER]: Refusing update from remote peer d896c150328d4d068bc22f2abdb68bc1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:41:23.284627 20087 consensus_queue.cc:1035] T ced525acd241456598612a11e487849f P d896c150328d4d068bc22f2abdb68bc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e8a5d3d63f4f7c8ae78c1eaf6caa27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:41:23.284825 20083 consensus_queue.cc:1035] T ced525acd241456598612a11e487849f P d896c150328d4d068bc22f2abdb68bc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39d1bf2b1e254c3987a4670cc0ea3d08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:41:23.290840 20090 mvcc.cc:205] Tried to move back new op lower bound from 7298580821124689920 to 7298580821041844224. Current Snapshot: MvccSnapshot[applied={T|T < 7298580821124689920}]
I20260619 14:41:23.291843 20091 mvcc.cc:205] Tried to move back new op lower bound from 7298580821124689920 to 7298580821041844224. Current Snapshot: MvccSnapshot[applied={T|T < 7298580821124689920}]
I20260619 14:41:23.292521 20092 mvcc.cc:205] Tried to move back new op lower bound from 7298580821124689920 to 7298580821041844224. Current Snapshot: MvccSnapshot[applied={T|T < 7298580821124689920}]
I20260619 14:41:23.301604 19600 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:41:23.301772 19600 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:41:23.304224 19600 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4e540ae00d824f739976958505f50fcf: Sending DeleteTablet for 3 replicas of tablet ced525acd241456598612a11e487849f
I20260619 14:41:23.305024 19994 tablet_service.cc:1514] Processing DeleteTablet for tablet ced525acd241456598612a11e487849f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:41:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33942
I20260619 14:41:23.305127 19860 tablet_service.cc:1514] Processing DeleteTablet for tablet ced525acd241456598612a11e487849f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:41:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37996
I20260619 14:41:23.305573 19726 tablet_service.cc:1514] Processing DeleteTablet for tablet ced525acd241456598612a11e487849f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:41:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33310
I20260619 14:41:23.305794 20100 tablet_replica.cc:330] T ced525acd241456598612a11e487849f P 39d1bf2b1e254c3987a4670cc0ea3d08: stopping tablet replica
I20260619 14:41:23.305917 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 19663
I20260619 14:41:23.306759 20100 raft_consensus.cc:2229] T ced525acd241456598612a11e487849f P 39d1bf2b1e254c3987a4670cc0ea3d08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:41:23.307099 20100 raft_consensus.cc:2258] T ced525acd241456598612a11e487849f P 39d1bf2b1e254c3987a4670cc0ea3d08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:41:23.308748 20099 tablet_replica.cc:330] T ced525acd241456598612a11e487849f P d896c150328d4d068bc22f2abdb68bc1: stopping tablet replica
I20260619 14:41:23.308763 20100 ts_tablet_manager.cc:1905] T ced525acd241456598612a11e487849f P 39d1bf2b1e254c3987a4670cc0ea3d08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:41:23.309116 20099 raft_consensus.cc:2229] T ced525acd241456598612a11e487849f P d896c150328d4d068bc22f2abdb68bc1 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:41:23.309574 20099 raft_consensus.cc:2258] T ced525acd241456598612a11e487849f P d896c150328d4d068bc22f2abdb68bc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:41:23.311115 20099 ts_tablet_manager.cc:1905] T ced525acd241456598612a11e487849f P d896c150328d4d068bc22f2abdb68bc1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:41:23.311492 20100 ts_tablet_manager.cc:1918] T ced525acd241456598612a11e487849f P 39d1bf2b1e254c3987a4670cc0ea3d08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:41:23.311587 20100 log.cc:1196] T ced525acd241456598612a11e487849f P 39d1bf2b1e254c3987a4670cc0ea3d08: Deleting WAL directory at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/ts-1/wal/wals/ced525acd241456598612a11e487849f
I20260619 14:41:23.311890 20100 ts_tablet_manager.cc:1939] T ced525acd241456598612a11e487849f P 39d1bf2b1e254c3987a4670cc0ea3d08: Deleting consensus metadata
I20260619 14:41:23.313174 19586 catalog_manager.cc:4841] TS 39d1bf2b1e254c3987a4670cc0ea3d08 (127.0.98.2:37033): tablet ced525acd241456598612a11e487849f (table test-table [id=9cdfbac9f0c44836af3db9891ad57173]) successfully deleted
W20260619 14:41:23.313469 19588 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37261 (error 108)
I20260619 14:41:23.313794 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 19797
I20260619 14:41:23.314741 20099 ts_tablet_manager.cc:1918] T ced525acd241456598612a11e487849f P d896c150328d4d068bc22f2abdb68bc1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:41:23.314903 20099 log.cc:1196] T ced525acd241456598612a11e487849f P d896c150328d4d068bc22f2abdb68bc1: Deleting WAL directory at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781879837648036-392-0/minicluster-data/ts-2/wal/wals/ced525acd241456598612a11e487849f
I20260619 14:41:23.315271 20099 ts_tablet_manager.cc:1939] T ced525acd241456598612a11e487849f P d896c150328d4d068bc22f2abdb68bc1: Deleting consensus metadata
I20260619 14:41:23.316318 20071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:23.316032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37261 (local address 127.0.0.1:33316)
0619 14:41:23.316150 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:23.316221 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260619 14:41:23.316431 19587 catalog_manager.cc:4841] TS d896c150328d4d068bc22f2abdb68bc1 (127.0.98.3:34521): tablet ced525acd241456598612a11e487849f (table test-table [id=9cdfbac9f0c44836af3db9891ad57173]) successfully deleted
W20260619 14:41:23.316457 19588 catalog_manager.cc:4584] TS a0e8a5d3d63f4f7c8ae78c1eaf6caa27 (127.0.98.1:37261): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ced525acd241456598612a11e487849f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37261: connect: Connection refused (error 111)
I20260619 14:41:23.319440 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 19931
I20260619 14:41:23.324136 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 19569
2026-06-19T14:41:23Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2164 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-19T14:41:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:41:23Z Disabled control of system clock
I20260619 14:41:23.351465 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41231
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41231
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 14:41:23.457068 20106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:41:23.457327 20106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:41:23.457360 20106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:41:23.461239 20106 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:41:23.461292 20106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:41:23.461306 20106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:41:23.461321 20106 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:41:23.461334 20106 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:41:23.465955 20106 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41231
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20106
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:41:23.466938 20106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:41:23.468227 20106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:41:23.476118 20114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:23.476118 20112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:23.476178 20111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:23.476420 20106 server_base.cc:1034] running on GCE node
I20260619 14:41:23.477051 20106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:41:23.478101 20106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:41:23.479286 20106 hybrid_clock.cc:648] HybridClock initialized: now 1781880083479266 us; error 26 us; skew 500 ppm
I20260619 14:41:23.481088 20106 webserver.cc:460] Webserver started at http://127.0.98.62:46491/ using document root <none> and password file <none>
I20260619 14:41:23.481609 20106 fs_manager.cc:359] Metadata directory not provided
I20260619 14:41:23.481680 20106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:41:23.481885 20106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:41:23.483827 20106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/master-0/data/instance:
uuid: "76c136ccc7a94f7a99a9e790687d6197"
format_stamp: "Formatted at 2026-06-19 14:41:23 on dist-test-slave-32hj"
server_key: "2ace12725ab7d5d6b11c24367034f0dc"
server_key_iv: "98e146f48d38e4fd9525a47306e7747c"
server_key_version: "encryptionkey@0"
I20260619 14:41:23.484372 20106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/master-0/wal/instance:
uuid: "76c136ccc7a94f7a99a9e790687d6197"
format_stamp: "Formatted at 2026-06-19 14:41:23 on dist-test-slave-32hj"
server_key: "2ace12725ab7d5d6b11c24367034f0dc"
server_key_iv: "98e146f48d38e4fd9525a47306e7747c"
server_key_version: "encryptionkey@0"
I20260619 14:41:23.487856 20106 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 14:41:23.490181 20120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:23.491310 20106 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:41:23.491417 20106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/master-0/wal
uuid: "76c136ccc7a94f7a99a9e790687d6197"
format_stamp: "Formatted at 2026-06-19 14:41:23 on dist-test-slave-32hj"
server_key: "2ace12725ab7d5d6b11c24367034f0dc"
server_key_iv: "98e146f48d38e4fd9525a47306e7747c"
server_key_version: "encryptionkey@0"
I20260619 14:41:23.491505 20106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:41:23.507120 20106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:41:23.507848 20106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:41:23.508016 20106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:41:23.514550 20106 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41231
I20260619 14:41:23.514572 20172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41231 every 8 connection(s)
I20260619 14:41:23.515453 20106 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:41:23.516417 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 20106
I20260619 14:41:23.516542 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/master-0/wal/instance
I20260619 14:41:23.516885 392 external_mini_cluster.cc:1428] Setting key 00e43858709dfffc9b360e1c5a1edaf6
I20260619 14:41:23.519035 20173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:41:23.524398 20173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76c136ccc7a94f7a99a9e790687d6197: Bootstrap starting.
I20260619 14:41:23.525916 20176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:23.518252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54500 (local address 127.0.98.62:41231)
0619 14:41:23.518927 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:23.518942 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:23.518979 (+ 37us) server_negotiation.cc:407] Connection header received
0619 14:41:23.519681 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:23.519700 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:23.520089 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:23.520406 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:23.520860 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:23.521817 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:23.522669 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:23.523133 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:23.523276 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:23.524127 (+ 851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:23.524133 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:23.524144 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:41:23.524547 (+ 403us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:23.524615 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:41:23.524637 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:23.524741 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:23.524876 (+ 135us) server_negotiation.cc:299] Negotiation successful
0619 14:41:23.525121 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":129,"threads_started":1}
I20260619 14:41:23.526705 20173 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76c136ccc7a94f7a99a9e790687d6197: Neither blocks nor log segments found. Creating new log.
I20260619 14:41:23.527415 20173 log.cc:824] T 00000000000000000000000000000000 P 76c136ccc7a94f7a99a9e790687d6197: Log is configured to *not* fsync() on all Append() calls
I20260619 14:41:23.529253 20173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76c136ccc7a94f7a99a9e790687d6197: No bootstrap required, opened a new log
I20260619 14:41:23.532497 20173 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76c136ccc7a94f7a99a9e790687d6197 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76c136ccc7a94f7a99a9e790687d6197" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41231 } }
I20260619 14:41:23.532708 20173 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76c136ccc7a94f7a99a9e790687d6197 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:41:23.532790 20173 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76c136ccc7a94f7a99a9e790687d6197 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c136ccc7a94f7a99a9e790687d6197, State: Initialized, Role: FOLLOWER
I20260619 14:41:23.533190 20173 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76c136ccc7a94f7a99a9e790687d6197 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76c136ccc7a94f7a99a9e790687d6197" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41231 } }
I20260619 14:41:23.533340 20173 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76c136ccc7a94f7a99a9e790687d6197 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:41:23.533408 20173 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76c136ccc7a94f7a99a9e790687d6197 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:41:23.533505 20173 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76c136ccc7a94f7a99a9e790687d6197 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:41:23.534444 20173 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76c136ccc7a94f7a99a9e790687d6197 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76c136ccc7a94f7a99a9e790687d6197" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41231 } }
I20260619 14:41:23.534744 20173 leader_election.cc:302] T 00000000000000000000000000000000 P 76c136ccc7a94f7a99a9e790687d6197 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76c136ccc7a94f7a99a9e790687d6197; no voters:
I20260619 14:41:23.535018 20173 leader_election.cc:288] T 00000000000000000000000000000000 P 76c136ccc7a94f7a99a9e790687d6197 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:41:23.535116 20178 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76c136ccc7a94f7a99a9e790687d6197 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:41:23.535378 20178 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76c136ccc7a94f7a99a9e790687d6197 [term 1 LEADER]: Becoming Leader. State: Replica: 76c136ccc7a94f7a99a9e790687d6197, State: Running, Role: LEADER
I20260619 14:41:23.535660 20178 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76c136ccc7a94f7a99a9e790687d6197 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76c136ccc7a94f7a99a9e790687d6197" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41231 } }
I20260619 14:41:23.536120 20173 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76c136ccc7a94f7a99a9e790687d6197 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:41:23.537081 20179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76c136ccc7a94f7a99a9e790687d6197 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76c136ccc7a94f7a99a9e790687d6197" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76c136ccc7a94f7a99a9e790687d6197" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41231 } } }
I20260619 14:41:23.537211 20179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76c136ccc7a94f7a99a9e790687d6197 [sys.catalog]: This master's current role is: LEADER
I20260619 14:41:23.537483 20180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76c136ccc7a94f7a99a9e790687d6197 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76c136ccc7a94f7a99a9e790687d6197. Latest consensus state: current_term: 1 leader_uuid: "76c136ccc7a94f7a99a9e790687d6197" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76c136ccc7a94f7a99a9e790687d6197" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41231 } } }
I20260619 14:41:23.537567 20180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76c136ccc7a94f7a99a9e790687d6197 [sys.catalog]: This master's current role is: LEADER
I20260619 14:41:23.537678 20187 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:41:23.540709 20187 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:41:23.545984 20187 catalog_manager.cc:1347] Generated new cluster ID: 87a9a0b9dbbf43baa614019612a7df48
I20260619 14:41:23.546034 20187 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:41:23.558185 20187 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:41:23.558985 20187 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:41:23.579731 20187 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76c136ccc7a94f7a99a9e790687d6197: Generated new TSK 0
I20260619 14:41:23.580560 20187 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:41:23.591558 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41231
--builtin_ntp_servers=127.0.98.20:45589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 14:41:23.695509 20197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:41:23.695755 20197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:41:23.695812 20197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:41:23.699688 20197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:41:23.699738 20197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:41:23.699808 20197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:41:23.704311 20197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41231
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:41:23.705283 20197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:41:23.706583 20197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:41:23.715619 20203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:23.715651 20202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:23.715703 20197 server_base.cc:1034] running on GCE node
W20260619 14:41:23.715639 20205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:23.716215 20197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:41:23.716849 20197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:41:23.717996 20197 hybrid_clock.cc:648] HybridClock initialized: now 1781880083717977 us; error 29 us; skew 500 ppm
I20260619 14:41:23.719735 20197 webserver.cc:460] Webserver started at http://127.0.98.1:35285/ using document root <none> and password file <none>
I20260619 14:41:23.720278 20197 fs_manager.cc:359] Metadata directory not provided
I20260619 14:41:23.720337 20197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:41:23.720528 20197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:41:23.722517 20197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/ts-0/data/instance:
uuid: "65a1a93520ed4f81a35dfaa5bb85c68f"
format_stamp: "Formatted at 2026-06-19 14:41:23 on dist-test-slave-32hj"
server_key: "9a47e1bb300566f99c2a8fb6d31db4bc"
server_key_iv: "7c22a352061d5427a1ec7b5e71882ba5"
server_key_version: "encryptionkey@0"
I20260619 14:41:23.723021 20197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/ts-0/wal/instance:
uuid: "65a1a93520ed4f81a35dfaa5bb85c68f"
format_stamp: "Formatted at 2026-06-19 14:41:23 on dist-test-slave-32hj"
server_key: "9a47e1bb300566f99c2a8fb6d31db4bc"
server_key_iv: "7c22a352061d5427a1ec7b5e71882ba5"
server_key_version: "encryptionkey@0"
I20260619 14:41:23.726603 20197 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260619 14:41:23.729054 20211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:23.730242 20197 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:41:23.730358 20197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/ts-0/wal
uuid: "65a1a93520ed4f81a35dfaa5bb85c68f"
format_stamp: "Formatted at 2026-06-19 14:41:23 on dist-test-slave-32hj"
server_key: "9a47e1bb300566f99c2a8fb6d31db4bc"
server_key_iv: "7c22a352061d5427a1ec7b5e71882ba5"
server_key_version: "encryptionkey@0"
I20260619 14:41:23.730453 20197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:41:23.751169 20197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:41:23.751853 20197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:41:23.752027 20197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:41:23.752583 20197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:41:23.753536 20197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:41:23.753582 20197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:23.753634 20197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:41:23.753664 20197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:23.761699 20197 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46545
I20260619 14:41:23.761734 20324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46545 every 8 connection(s)
I20260619 14:41:23.762560 20197 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:41:23.767124 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 20197
I20260619 14:41:23.767264 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:41:23.767575 392 external_mini_cluster.cc:1428] Setting key b06dcb911a2f4cd3b600a59cf9379e96
I20260619 14:41:23.769639 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41231
--builtin_ntp_servers=127.0.98.20:45589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 14:41:23.772289 20176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:23.764217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36585 (local address 127.0.98.62:41231)
0619 14:41:23.764347 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:23.764351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:23.765354 (+ 1003us) server_negotiation.cc:407] Connection header received
0619 14:41:23.766629 (+ 1275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:23.766632 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:23.766678 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:23.766748 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:23.767873 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:23.768786 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:23.770226 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:23.770524 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:23.770637 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:23.771864 (+ 1227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:23.771867 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:23.771869 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:41:23.771880 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:23.771901 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:41:23.771909 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:23.771965 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:23.772116 (+ 151us) server_negotiation.cc:299] Negotiation successful
0619 14:41:23.772181 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 14:41:23.773423 20327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:23.764585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41231 (local address 127.0.98.1:36585)
0619 14:41:23.765228 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:23.765270 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:23.766460 (+ 1190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:23.766918 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:23.766926 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:23.767030 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:41:23.767684 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:23.767704 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:23.768941 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:23.768945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:23.770114 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:23.770124 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:23.770697 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:23.770700 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:23.770943 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:23.771563 (+ 620us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:41:23.771566 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:41:23.771620 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:41:23.771702 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:41:23.771985 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:41:23.771993 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:41:23.771998 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 14:41:23.772117 (+ 119us) client_negotiation.cc:240] Negotiation successful
0619 14:41:23.772421 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":139,"threads_started":1}
I20260619 14:41:23.774617 20325 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41231
I20260619 14:41:23.774832 20325 heartbeater.cc:464] Registering TS with master...
I20260619 14:41:23.775303 20325 heartbeater.cc:511] Master 127.0.98.62:41231 requested a full tablet report, sending...
I20260619 14:41:23.776688 20137 ts_manager.cc:194] Registered new tserver with Master: 65a1a93520ed4f81a35dfaa5bb85c68f (127.0.98.1:46545)
I20260619 14:41:23.777820 20137 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:36585
W20260619 14:41:23.875805 20328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:41:23.876060 20328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:41:23.876097 20328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:41:23.880168 20328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:41:23.880228 20328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:41:23.880285 20328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:41:23.884954 20328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41231
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:41:23.885982 20328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:41:23.887338 20328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:41:23.896564 20334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:23.896559 20333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:23.896649 20328 server_base.cc:1034] running on GCE node
W20260619 14:41:23.896551 20336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:23.897154 20328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:41:23.897755 20328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:41:23.898922 20328 hybrid_clock.cc:648] HybridClock initialized: now 1781880083898898 us; error 36 us; skew 500 ppm
I20260619 14:41:23.900746 20328 webserver.cc:460] Webserver started at http://127.0.98.2:37289/ using document root <none> and password file <none>
I20260619 14:41:23.901292 20328 fs_manager.cc:359] Metadata directory not provided
I20260619 14:41:23.901351 20328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:41:23.901539 20328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:41:23.903503 20328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/ts-1/data/instance:
uuid: "d406f7ab5f5a41ab80213fbb90a5ed3c"
format_stamp: "Formatted at 2026-06-19 14:41:23 on dist-test-slave-32hj"
server_key: "9d9eac3352645a1103b043edb1331c56"
server_key_iv: "56806b3bbf965ff11f5c3c9f32cffbce"
server_key_version: "encryptionkey@0"
I20260619 14:41:23.904029 20328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d406f7ab5f5a41ab80213fbb90a5ed3c"
format_stamp: "Formatted at 2026-06-19 14:41:23 on dist-test-slave-32hj"
server_key: "9d9eac3352645a1103b043edb1331c56"
server_key_iv: "56806b3bbf965ff11f5c3c9f32cffbce"
server_key_version: "encryptionkey@0"
I20260619 14:41:23.907466 20328 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 14:41:23.909797 20342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:23.911015 20328 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:41:23.911115 20328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/ts-1/wal
uuid: "d406f7ab5f5a41ab80213fbb90a5ed3c"
format_stamp: "Formatted at 2026-06-19 14:41:23 on dist-test-slave-32hj"
server_key: "9d9eac3352645a1103b043edb1331c56"
server_key_iv: "56806b3bbf965ff11f5c3c9f32cffbce"
server_key_version: "encryptionkey@0"
I20260619 14:41:23.911209 20328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:41:23.933216 20328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:41:23.933876 20328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:41:23.934054 20328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:41:23.934629 20328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:41:23.935564 20328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:41:23.935621 20328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:23.935693 20328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:41:23.935721 20328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:23.944442 20328 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38739
I20260619 14:41:23.944504 20455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38739 every 8 connection(s)
I20260619 14:41:23.945425 20328 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:41:23.945827 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 20328
I20260619 14:41:23.945963 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:41:23.946298 392 external_mini_cluster.cc:1428] Setting key b7b48619784e703b299a69c79b19367c
I20260619 14:41:23.948498 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41231
--builtin_ntp_servers=127.0.98.20:45589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 14:41:23.955799 20176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:23.947489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33469 (local address 127.0.98.62:41231)
0619 14:41:23.947639 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:23.947644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:23.949032 (+ 1388us) server_negotiation.cc:407] Connection header received
0619 14:41:23.950677 (+ 1645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:23.950680 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:23.950735 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:23.950829 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:23.951713 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:23.952528 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:23.953753 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:23.954089 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:23.954221 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:23.955297 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:23.955301 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:23.955304 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:41:23.955320 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:23.955345 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:41:23.955356 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:23.955436 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:23.955549 (+ 113us) server_negotiation.cc:299] Negotiation successful
0619 14:41:23.955618 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 14:41:23.956676 20458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:23.948034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41231 (local address 127.0.98.2:33469)
0619 14:41:23.948876 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:23.948932 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:23.950447 (+ 1515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:23.950975 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:23.950981 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:23.951068 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:41:23.951580 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:23.951594 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:23.952670 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:23.952674 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:23.953635 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:23.953645 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:23.954244 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:23.954264 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:23.954410 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:23.955035 (+ 625us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:41:23.955036 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:41:23.955079 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:41:23.955145 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:41:23.955431 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:41:23.955435 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:41:23.955439 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 14:41:23.955529 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 14:41:23.955763 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":130,"threads_started":1}
I20260619 14:41:23.957800 20456 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41231
I20260619 14:41:23.958072 20456 heartbeater.cc:464] Registering TS with master...
I20260619 14:41:23.958714 20456 heartbeater.cc:511] Master 127.0.98.62:41231 requested a full tablet report, sending...
I20260619 14:41:23.959847 20137 ts_manager.cc:194] Registered new tserver with Master: d406f7ab5f5a41ab80213fbb90a5ed3c (127.0.98.2:38739)
I20260619 14:41:23.960414 20137 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:33469
W20260619 14:41:24.066303 20459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:41:24.066562 20459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:41:24.066601 20459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:41:24.070662 20459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:41:24.070724 20459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:41:24.070779 20459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:41:24.075589 20459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41231
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:41:24.076639 20459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:41:24.078033 20459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:41:24.087347 20465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:24.087404 20459 server_base.cc:1034] running on GCE node
W20260619 14:41:24.087337 20464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:24.087352 20467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:24.088019 20459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:41:24.088636 20459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:41:24.089814 20459 hybrid_clock.cc:648] HybridClock initialized: now 1781880084089795 us; error 28 us; skew 500 ppm
I20260619 14:41:24.091689 20459 webserver.cc:460] Webserver started at http://127.0.98.3:44269/ using document root <none> and password file <none>
I20260619 14:41:24.092252 20459 fs_manager.cc:359] Metadata directory not provided
I20260619 14:41:24.092319 20459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:41:24.092515 20459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:41:24.094659 20459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/ts-2/data/instance:
uuid: "2328bfd725404468be67f4f08fec6590"
format_stamp: "Formatted at 2026-06-19 14:41:24 on dist-test-slave-32hj"
server_key: "e4de2649bd1d3f88c22eeb908f7cc6b0"
server_key_iv: "13adea11821b207820b3439d2dfd1901"
server_key_version: "encryptionkey@0"
I20260619 14:41:24.095186 20459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2328bfd725404468be67f4f08fec6590"
format_stamp: "Formatted at 2026-06-19 14:41:24 on dist-test-slave-32hj"
server_key: "e4de2649bd1d3f88c22eeb908f7cc6b0"
server_key_iv: "13adea11821b207820b3439d2dfd1901"
server_key_version: "encryptionkey@0"
I20260619 14:41:24.098781 20459 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:41:24.101291 20473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:24.102545 20459 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:41:24.102658 20459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/ts-2/wal
uuid: "2328bfd725404468be67f4f08fec6590"
format_stamp: "Formatted at 2026-06-19 14:41:24 on dist-test-slave-32hj"
server_key: "e4de2649bd1d3f88c22eeb908f7cc6b0"
server_key_iv: "13adea11821b207820b3439d2dfd1901"
server_key_version: "encryptionkey@0"
I20260619 14:41:24.102756 20459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:41:24.121868 20459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:41:24.122575 20459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:41:24.122746 20459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:41:24.123325 20459 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:41:24.124258 20459 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:41:24.124305 20459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:24.124356 20459 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:41:24.124382 20459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:24.133301 20459 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40377
I20260619 14:41:24.133353 20586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40377 every 8 connection(s)
I20260619 14:41:24.134296 20459 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:41:24.134723 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 20459
I20260619 14:41:24.134843 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:41:24.135147 392 external_mini_cluster.cc:1428] Setting key cef40c63973715a2e804c1baa556ec9a
I20260619 14:41:24.143271 20176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:24.136192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54729 (local address 127.0.98.62:41231)
0619 14:41:24.136332 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:24.136337 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:24.137649 (+ 1312us) server_negotiation.cc:407] Connection header received
0619 14:41:24.139051 (+ 1402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:24.139054 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:24.139098 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:24.139163 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:24.140002 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:24.140525 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:24.141647 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:24.141870 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:24.141949 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:24.142905 (+ 956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:24.142908 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:24.142910 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:41:24.142922 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:24.142940 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:41:24.142948 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:24.143002 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:24.143105 (+ 103us) server_negotiation.cc:299] Negotiation successful
0619 14:41:24.143155 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 14:41:24.144137 20589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:24.136712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41231 (local address 127.0.98.3:54729)
0619 14:41:24.137518 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:24.137570 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:24.138864 (+ 1294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:24.139302 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:24.139308 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:24.139395 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:41:24.139881 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:24.139893 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:24.140613 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:24.140615 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:24.141545 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:24.141554 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:24.141986 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:24.141989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:24.142123 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:24.142672 (+ 549us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:41:24.142674 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:41:24.142713 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:41:24.142777 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:41:24.143007 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:41:24.143011 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:41:24.143015 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 14:41:24.143101 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 14:41:24.143339 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":128,"threads_started":1}
I20260619 14:41:24.145321 20587 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41231
I20260619 14:41:24.145579 20587 heartbeater.cc:464] Registering TS with master...
I20260619 14:41:24.146095 20587 heartbeater.cc:511] Master 127.0.98.62:41231 requested a full tablet report, sending...
I20260619 14:41:24.147133 20137 ts_manager.cc:194] Registered new tserver with Master: 2328bfd725404468be67f4f08fec6590 (127.0.98.3:40377)
I20260619 14:41:24.147692 20137 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:54729
I20260619 14:41:24.149158 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:41:24.154421 20176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:24.150981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54508 (local address 127.0.98.62:41231)
0619 14:41:24.151105 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:24.151109 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:24.151295 (+ 186us) server_negotiation.cc:407] Connection header received
0619 14:41:24.151385 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:24.151388 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:24.151426 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:24.151487 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:24.151690 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:24.152186 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:24.153037 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:24.153255 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:24.153342 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:24.154061 (+ 719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:24.154064 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:24.154066 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:41:24.154077 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:24.154095 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:41:24.154103 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:24.154166 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:24.154225 (+ 59us) server_negotiation.cc:299] Negotiation successful
0619 14:41:24.154278 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 14:41:24.157110 20137 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54508:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:41:24.160113 20137 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:41:24.173619 20601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:24.167883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46545 (local address 127.0.0.1:57028)
0619 14:41:24.168711 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:24.168783 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:24.168915 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:24.169613 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:24.169619 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:24.169640 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:41:24.169779 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:24.169786 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:24.170738 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:24.170741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:24.172090 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:24.172100 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:24.173090 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:24.173093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:24.173199 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:24.173235 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 14:41:24.173342 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 14:41:24.173458 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"mutex_wait_us":33,"spinlock_wait_cycles":6272,"thread_start_us":63,"threads_started":1}
I20260619 14:41:24.173621 20600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:24.168004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57028 (local address 127.0.98.1:46545)
0619 14:41:24.169180 (+ 1176us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:24.169191 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:24.169214 (+ 23us) server_negotiation.cc:407] Connection header received
0619 14:41:24.169304 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:24.169308 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:24.169465 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:24.169597 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:41:24.169898 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:24.170631 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:24.172214 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:24.172986 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:24.173077 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:24.173275 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:24.173370 (+ 95us) server_negotiation.cc:299] Negotiation successful
0619 14:41:24.173459 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1076,"thread_start_us":122,"threads_started":1}
I20260619 14:41:24.174201 20598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:24.168260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38739 (local address 127.0.0.1:45406)
0619 14:41:24.169113 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:24.169124 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:24.169191 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:24.170208 (+ 1017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:24.170211 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:24.170227 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:41:24.170310 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:24.170316 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:24.171512 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:24.171516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:24.172487 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:24.172495 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:24.173675 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:24.173822 (+ 147us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:24.173908 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:24.173932 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 14:41:24.174005 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 14:41:24.174068 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":800,"mutex_wait_us":10,"thread_start_us":87,"threads_started":1}
I20260619 14:41:24.174300 20602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:24.168484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45406 (local address 127.0.98.2:38739)
0619 14:41:24.169882 (+ 1398us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:24.169897 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:24.169916 (+ 19us) server_negotiation.cc:407] Connection header received
0619 14:41:24.169970 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:24.169974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:24.170090 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:24.170202 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:41:24.170407 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:24.171389 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:24.172596 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:24.173328 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:24.173431 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:24.173609 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:24.174053 (+ 444us) server_negotiation.cc:299] Negotiation successful
0619 14:41:24.174151 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1302,"thread_start_us":143,"threads_started":1}
I20260619 14:41:24.174501 20599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:24.168018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40377 (local address 127.0.0.1:34526)
0619 14:41:24.168711 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:24.168783 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:24.168915 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:24.169661 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:24.169664 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:24.169682 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:41:24.169786 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:24.169793 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:24.171690 (+ 1897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:24.171693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:24.172687 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:24.172697 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:24.173951 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:24.173954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:24.174033 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:24.174055 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 14:41:24.174103 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 14:41:24.174154 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"spinlock_wait_cycles":5248,"thread_start_us":105,"threads_started":1}
I20260619 14:41:24.175055 20603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:24.168857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34526 (local address 127.0.98.3:40377)
0619 14:41:24.169308 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:24.169317 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:24.169337 (+ 20us) server_negotiation.cc:407] Connection header received
0619 14:41:24.169398 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:24.169402 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:24.169540 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:24.169669 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:41:24.170760 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:24.171576 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:24.172829 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:24.173580 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:24.173683 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:24.174817 (+ 1134us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:24.174858 (+ 41us) server_negotiation.cc:299] Negotiation successful
0619 14:41:24.174929 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":84,"threads_started":1}
I20260619 14:41:24.176633 20259 tablet_service.cc:1467] Processing CreateTablet for tablet 7cd3b34b65f14fdda9728eda8d3672e5 (DEFAULT_TABLE table=test-table [id=1f2d26112e7045e6bc5b24baee9bfbde]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:41:24.177059 20390 tablet_service.cc:1467] Processing CreateTablet for tablet 7cd3b34b65f14fdda9728eda8d3672e5 (DEFAULT_TABLE table=test-table [id=1f2d26112e7045e6bc5b24baee9bfbde]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:41:24.177590 20259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cd3b34b65f14fdda9728eda8d3672e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:41:24.177418 20521 tablet_service.cc:1467] Processing CreateTablet for tablet 7cd3b34b65f14fdda9728eda8d3672e5 (DEFAULT_TABLE table=test-table [id=1f2d26112e7045e6bc5b24baee9bfbde]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:41:24.178049 20390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cd3b34b65f14fdda9728eda8d3672e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:41:24.178261 20521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cd3b34b65f14fdda9728eda8d3672e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:41:24.183281 20604 tablet_bootstrap.cc:492] T 7cd3b34b65f14fdda9728eda8d3672e5 P 65a1a93520ed4f81a35dfaa5bb85c68f: Bootstrap starting.
I20260619 14:41:24.185083 20605 tablet_bootstrap.cc:492] T 7cd3b34b65f14fdda9728eda8d3672e5 P d406f7ab5f5a41ab80213fbb90a5ed3c: Bootstrap starting.
I20260619 14:41:24.185408 20606 tablet_bootstrap.cc:492] T 7cd3b34b65f14fdda9728eda8d3672e5 P 2328bfd725404468be67f4f08fec6590: Bootstrap starting.
I20260619 14:41:24.186415 20604 tablet_bootstrap.cc:654] T 7cd3b34b65f14fdda9728eda8d3672e5 P 65a1a93520ed4f81a35dfaa5bb85c68f: Neither blocks nor log segments found. Creating new log.
I20260619 14:41:24.187042 20606 tablet_bootstrap.cc:654] T 7cd3b34b65f14fdda9728eda8d3672e5 P 2328bfd725404468be67f4f08fec6590: Neither blocks nor log segments found. Creating new log.
I20260619 14:41:24.187134 20605 tablet_bootstrap.cc:654] T 7cd3b34b65f14fdda9728eda8d3672e5 P d406f7ab5f5a41ab80213fbb90a5ed3c: Neither blocks nor log segments found. Creating new log.
I20260619 14:41:24.187263 20604 log.cc:824] T 7cd3b34b65f14fdda9728eda8d3672e5 P 65a1a93520ed4f81a35dfaa5bb85c68f: Log is configured to *not* fsync() on all Append() calls
I20260619 14:41:24.187755 20606 log.cc:824] T 7cd3b34b65f14fdda9728eda8d3672e5 P 2328bfd725404468be67f4f08fec6590: Log is configured to *not* fsync() on all Append() calls
I20260619 14:41:24.187894 20605 log.cc:824] T 7cd3b34b65f14fdda9728eda8d3672e5 P d406f7ab5f5a41ab80213fbb90a5ed3c: Log is configured to *not* fsync() on all Append() calls
I20260619 14:41:24.189358 20604 tablet_bootstrap.cc:492] T 7cd3b34b65f14fdda9728eda8d3672e5 P 65a1a93520ed4f81a35dfaa5bb85c68f: No bootstrap required, opened a new log
I20260619 14:41:24.189541 20605 tablet_bootstrap.cc:492] T 7cd3b34b65f14fdda9728eda8d3672e5 P d406f7ab5f5a41ab80213fbb90a5ed3c: No bootstrap required, opened a new log
I20260619 14:41:24.189637 20604 ts_tablet_manager.cc:1397] T 7cd3b34b65f14fdda9728eda8d3672e5 P 65a1a93520ed4f81a35dfaa5bb85c68f: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 14:41:24.189663 20606 tablet_bootstrap.cc:492] T 7cd3b34b65f14fdda9728eda8d3672e5 P 2328bfd725404468be67f4f08fec6590: No bootstrap required, opened a new log
I20260619 14:41:24.189721 20605 ts_tablet_manager.cc:1397] T 7cd3b34b65f14fdda9728eda8d3672e5 P d406f7ab5f5a41ab80213fbb90a5ed3c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:41:24.189810 20606 ts_tablet_manager.cc:1397] T 7cd3b34b65f14fdda9728eda8d3672e5 P 2328bfd725404468be67f4f08fec6590: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 14:41:24.192961 20605 raft_consensus.cc:348] T 7cd3b34b65f14fdda9728eda8d3672e5 P d406f7ab5f5a41ab80213fbb90a5ed3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65a1a93520ed4f81a35dfaa5bb85c68f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46545 } } peers { permanent_uuid: "2328bfd725404468be67f4f08fec6590" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40377 } } peers { permanent_uuid: "d406f7ab5f5a41ab80213fbb90a5ed3c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38739 } }
I20260619 14:41:24.193182 20605 raft_consensus.cc:374] T 7cd3b34b65f14fdda9728eda8d3672e5 P d406f7ab5f5a41ab80213fbb90a5ed3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:41:24.193233 20605 raft_consensus.cc:729] T 7cd3b34b65f14fdda9728eda8d3672e5 P d406f7ab5f5a41ab80213fbb90a5ed3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d406f7ab5f5a41ab80213fbb90a5ed3c, State: Initialized, Role: FOLLOWER
I20260619 14:41:24.193640 20605 consensus_queue.cc:260] T 7cd3b34b65f14fdda9728eda8d3672e5 P d406f7ab5f5a41ab80213fbb90a5ed3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65a1a93520ed4f81a35dfaa5bb85c68f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46545 } } peers { permanent_uuid: "2328bfd725404468be67f4f08fec6590" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40377 } } peers { permanent_uuid: "d406f7ab5f5a41ab80213fbb90a5ed3c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38739 } }
I20260619 14:41:24.193873 20606 raft_consensus.cc:348] T 7cd3b34b65f14fdda9728eda8d3672e5 P 2328bfd725404468be67f4f08fec6590 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65a1a93520ed4f81a35dfaa5bb85c68f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46545 } } peers { permanent_uuid: "2328bfd725404468be67f4f08fec6590" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40377 } } peers { permanent_uuid: "d406f7ab5f5a41ab80213fbb90a5ed3c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38739 } }
I20260619 14:41:24.193873 20604 raft_consensus.cc:348] T 7cd3b34b65f14fdda9728eda8d3672e5 P 65a1a93520ed4f81a35dfaa5bb85c68f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65a1a93520ed4f81a35dfaa5bb85c68f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46545 } } peers { permanent_uuid: "2328bfd725404468be67f4f08fec6590" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40377 } } peers { permanent_uuid: "d406f7ab5f5a41ab80213fbb90a5ed3c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38739 } }
I20260619 14:41:24.194120 20604 raft_consensus.cc:374] T 7cd3b34b65f14fdda9728eda8d3672e5 P 65a1a93520ed4f81a35dfaa5bb85c68f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:41:24.194120 20606 raft_consensus.cc:374] T 7cd3b34b65f14fdda9728eda8d3672e5 P 2328bfd725404468be67f4f08fec6590 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:41:24.194175 20606 raft_consensus.cc:729] T 7cd3b34b65f14fdda9728eda8d3672e5 P 2328bfd725404468be67f4f08fec6590 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2328bfd725404468be67f4f08fec6590, State: Initialized, Role: FOLLOWER
I20260619 14:41:24.194175 20604 raft_consensus.cc:729] T 7cd3b34b65f14fdda9728eda8d3672e5 P 65a1a93520ed4f81a35dfaa5bb85c68f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65a1a93520ed4f81a35dfaa5bb85c68f, State: Initialized, Role: FOLLOWER
I20260619 14:41:24.194428 20456 heartbeater.cc:503] Master 127.0.98.62:41231 was elected leader, sending a full tablet report...
I20260619 14:41:24.194598 20606 consensus_queue.cc:260] T 7cd3b34b65f14fdda9728eda8d3672e5 P 2328bfd725404468be67f4f08fec6590 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65a1a93520ed4f81a35dfaa5bb85c68f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46545 } } peers { permanent_uuid: "2328bfd725404468be67f4f08fec6590" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40377 } } peers { permanent_uuid: "d406f7ab5f5a41ab80213fbb90a5ed3c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38739 } }
I20260619 14:41:24.194598 20604 consensus_queue.cc:260] T 7cd3b34b65f14fdda9728eda8d3672e5 P 65a1a93520ed4f81a35dfaa5bb85c68f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65a1a93520ed4f81a35dfaa5bb85c68f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46545 } } peers { permanent_uuid: "2328bfd725404468be67f4f08fec6590" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40377 } } peers { permanent_uuid: "d406f7ab5f5a41ab80213fbb90a5ed3c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38739 } }
I20260619 14:41:24.194744 20605 ts_tablet_manager.cc:1428] T 7cd3b34b65f14fdda9728eda8d3672e5 P d406f7ab5f5a41ab80213fbb90a5ed3c: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 14:41:24.195505 20604 ts_tablet_manager.cc:1428] T 7cd3b34b65f14fdda9728eda8d3672e5 P 65a1a93520ed4f81a35dfaa5bb85c68f: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 14:41:24.195505 20606 ts_tablet_manager.cc:1428] T 7cd3b34b65f14fdda9728eda8d3672e5 P 2328bfd725404468be67f4f08fec6590: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 14:41:24.195875 20325 heartbeater.cc:503] Master 127.0.98.62:41231 was elected leader, sending a full tablet report...
I20260619 14:41:24.196018 20587 heartbeater.cc:503] Master 127.0.98.62:41231 was elected leader, sending a full tablet report...
W20260619 14:41:24.196398 20457 tablet.cc:2369] T 7cd3b34b65f14fdda9728eda8d3672e5 P d406f7ab5f5a41ab80213fbb90a5ed3c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:41:24.263760 20326 tablet.cc:2369] T 7cd3b34b65f14fdda9728eda8d3672e5 P 65a1a93520ed4f81a35dfaa5bb85c68f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:41:24.385377 20588 tablet.cc:2369] T 7cd3b34b65f14fdda9728eda8d3672e5 P 2328bfd725404468be67f4f08fec6590: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:41:24.458474 20612 raft_consensus.cc:482] T 7cd3b34b65f14fdda9728eda8d3672e5 P 65a1a93520ed4f81a35dfaa5bb85c68f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:41:24.458673 20612 raft_consensus.cc:504] T 7cd3b34b65f14fdda9728eda8d3672e5 P 65a1a93520ed4f81a35dfaa5bb85c68f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65a1a93520ed4f81a35dfaa5bb85c68f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46545 } } peers { permanent_uuid: "2328bfd725404468be67f4f08fec6590" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40377 } } peers { permanent_uuid: "d406f7ab5f5a41ab80213fbb90a5ed3c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38739 } }
I20260619 14:41:24.459750 20612 leader_election.cc:288] T 7cd3b34b65f14fdda9728eda8d3672e5 P 65a1a93520ed4f81a35dfaa5bb85c68f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2328bfd725404468be67f4f08fec6590 (127.0.98.3:40377), d406f7ab5f5a41ab80213fbb90a5ed3c (127.0.98.2:38739)
I20260619 14:41:24.464823 20602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:24.460080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47143 (local address 127.0.98.2:38739)
0619 14:41:24.460272 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:24.460277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:24.460359 (+ 82us) server_negotiation.cc:407] Connection header received
0619 14:41:24.460506 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:24.460509 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:24.460556 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:24.460637 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:41:24.461020 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:24.461814 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:24.463526 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:24.464145 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:24.464253 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:24.464318 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:24.464584 (+ 266us) server_negotiation.cc:299] Negotiation successful
0619 14:41:24.464679 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260619 14:41:24.464823 20603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:24.460120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46967 (local address 127.0.98.3:40377)
0619 14:41:24.460279 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:24.460284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:24.460357 (+ 73us) server_negotiation.cc:407] Connection header received
0619 14:41:24.460507 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:24.460509 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:24.460556 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:24.460637 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:41:24.461016 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:24.461815 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:24.463526 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:24.464145 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:24.464278 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:24.464329 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:24.464584 (+ 255us) server_negotiation.cc:299] Negotiation successful
0619 14:41:24.464671 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 14:41:24.464840 20613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:24.459949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40377 (local address 127.0.98.1:46967)
0619 14:41:24.460270 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:24.460284 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:24.460411 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:24.460654 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:24.460657 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:24.460678 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:41:24.460917 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:24.460922 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:24.461928 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:24.461932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:24.463388 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:24.463400 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:24.464262 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:24.464265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:24.464415 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:24.464466 (+ 51us) client_negotiation.cc:769] Sending connection context
0619 14:41:24.464529 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 14:41:24.464622 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":181,"spinlock_wait_cycles":9728,"thread_start_us":93,"threads_started":1}
I20260619 14:41:24.464843 20614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:24.459994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38739 (local address 127.0.98.1:47143)
0619 14:41:24.460284 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:24.460295 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:24.460411 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:24.460668 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:24.460670 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:24.460687 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:41:24.460909 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:24.460917 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:24.461928 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:24.461932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:24.463388 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:24.463400 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:24.464262 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:24.464265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:24.464420 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:24.464454 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 14:41:24.464529 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 14:41:24.464622 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":214,"spinlock_wait_cycles":21760,"thread_start_us":62,"threads_started":1}
I20260619 14:41:24.465492 20541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cd3b34b65f14fdda9728eda8d3672e5" candidate_uuid: "65a1a93520ed4f81a35dfaa5bb85c68f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2328bfd725404468be67f4f08fec6590" is_pre_election: true
I20260619 14:41:24.465613 20410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cd3b34b65f14fdda9728eda8d3672e5" candidate_uuid: "65a1a93520ed4f81a35dfaa5bb85c68f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d406f7ab5f5a41ab80213fbb90a5ed3c" is_pre_election: true
I20260619 14:41:24.465795 20541 raft_consensus.cc:2454] T 7cd3b34b65f14fdda9728eda8d3672e5 P 2328bfd725404468be67f4f08fec6590 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65a1a93520ed4f81a35dfaa5bb85c68f in term 0.
I20260619 14:41:24.465891 20410 raft_consensus.cc:2454] T 7cd3b34b65f14fdda9728eda8d3672e5 P d406f7ab5f5a41ab80213fbb90a5ed3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65a1a93520ed4f81a35dfaa5bb85c68f in term 0.
I20260619 14:41:24.466276 20215 leader_election.cc:302] T 7cd3b34b65f14fdda9728eda8d3672e5 P 65a1a93520ed4f81a35dfaa5bb85c68f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2328bfd725404468be67f4f08fec6590, 65a1a93520ed4f81a35dfaa5bb85c68f, d406f7ab5f5a41ab80213fbb90a5ed3c; no voters:
I20260619 14:41:24.466560 20612 raft_consensus.cc:2781] T 7cd3b34b65f14fdda9728eda8d3672e5 P 65a1a93520ed4f81a35dfaa5bb85c68f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:41:24.466639 20612 raft_consensus.cc:482] T 7cd3b34b65f14fdda9728eda8d3672e5 P 65a1a93520ed4f81a35dfaa5bb85c68f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:41:24.466691 20612 raft_consensus.cc:3037] T 7cd3b34b65f14fdda9728eda8d3672e5 P 65a1a93520ed4f81a35dfaa5bb85c68f [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:41:24.467680 20612 raft_consensus.cc:504] T 7cd3b34b65f14fdda9728eda8d3672e5 P 65a1a93520ed4f81a35dfaa5bb85c68f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65a1a93520ed4f81a35dfaa5bb85c68f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46545 } } peers { permanent_uuid: "2328bfd725404468be67f4f08fec6590" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40377 } } peers { permanent_uuid: "d406f7ab5f5a41ab80213fbb90a5ed3c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38739 } }
I20260619 14:41:24.468010 20612 leader_election.cc:288] T 7cd3b34b65f14fdda9728eda8d3672e5 P 65a1a93520ed4f81a35dfaa5bb85c68f [CANDIDATE]: Term 1 election: Requested vote from peers 2328bfd725404468be67f4f08fec6590 (127.0.98.3:40377), d406f7ab5f5a41ab80213fbb90a5ed3c (127.0.98.2:38739)
I20260619 14:41:24.468318 20541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cd3b34b65f14fdda9728eda8d3672e5" candidate_uuid: "65a1a93520ed4f81a35dfaa5bb85c68f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2328bfd725404468be67f4f08fec6590"
I20260619 14:41:24.468343 20410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cd3b34b65f14fdda9728eda8d3672e5" candidate_uuid: "65a1a93520ed4f81a35dfaa5bb85c68f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d406f7ab5f5a41ab80213fbb90a5ed3c"
I20260619 14:41:24.468433 20541 raft_consensus.cc:3037] T 7cd3b34b65f14fdda9728eda8d3672e5 P 2328bfd725404468be67f4f08fec6590 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:41:24.468437 20410 raft_consensus.cc:3037] T 7cd3b34b65f14fdda9728eda8d3672e5 P d406f7ab5f5a41ab80213fbb90a5ed3c [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:41:24.469408 20410 raft_consensus.cc:2454] T 7cd3b34b65f14fdda9728eda8d3672e5 P d406f7ab5f5a41ab80213fbb90a5ed3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65a1a93520ed4f81a35dfaa5bb85c68f in term 1.
I20260619 14:41:24.469451 20541 raft_consensus.cc:2454] T 7cd3b34b65f14fdda9728eda8d3672e5 P 2328bfd725404468be67f4f08fec6590 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65a1a93520ed4f81a35dfaa5bb85c68f in term 1.
I20260619 14:41:24.469688 20212 leader_election.cc:302] T 7cd3b34b65f14fdda9728eda8d3672e5 P 65a1a93520ed4f81a35dfaa5bb85c68f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65a1a93520ed4f81a35dfaa5bb85c68f, d406f7ab5f5a41ab80213fbb90a5ed3c; no voters:
I20260619 14:41:24.469887 20612 raft_consensus.cc:2781] T 7cd3b34b65f14fdda9728eda8d3672e5 P 65a1a93520ed4f81a35dfaa5bb85c68f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:41:24.470057 20612 raft_consensus.cc:686] T 7cd3b34b65f14fdda9728eda8d3672e5 P 65a1a93520ed4f81a35dfaa5bb85c68f [term 1 LEADER]: Becoming Leader. State: Replica: 65a1a93520ed4f81a35dfaa5bb85c68f, State: Running, Role: LEADER
I20260619 14:41:24.470376 20612 consensus_queue.cc:237] T 7cd3b34b65f14fdda9728eda8d3672e5 P 65a1a93520ed4f81a35dfaa5bb85c68f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65a1a93520ed4f81a35dfaa5bb85c68f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46545 } } peers { permanent_uuid: "2328bfd725404468be67f4f08fec6590" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40377 } } peers { permanent_uuid: "d406f7ab5f5a41ab80213fbb90a5ed3c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38739 } }
I20260619 14:41:24.473362 20134 catalog_manager.cc:5495] T 7cd3b34b65f14fdda9728eda8d3672e5 P 65a1a93520ed4f81a35dfaa5bb85c68f reported cstate change: term changed from 0 to 1, leader changed from <none> to 65a1a93520ed4f81a35dfaa5bb85c68f (127.0.98.1). New cstate: current_term: 1 leader_uuid: "65a1a93520ed4f81a35dfaa5bb85c68f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65a1a93520ed4f81a35dfaa5bb85c68f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2328bfd725404468be67f4f08fec6590" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d406f7ab5f5a41ab80213fbb90a5ed3c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38739 } health_report { overall_health: UNKNOWN } } }
I20260619 14:41:24.536077 20600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:24.532984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57038 (local address 127.0.98.1:46545)
0619 14:41:24.533101 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:24.533104 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:24.533116 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:41:24.533207 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:24.533209 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:24.533269 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:24.533355 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:41:24.533533 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:24.534076 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:24.534885 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:24.535087 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:24.535154 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:24.535364 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:41:24.535776 (+ 412us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:41:24.535861 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:24.535913 (+ 52us) server_negotiation.cc:299] Negotiation successful
0619 14:41:24.535959 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 14:41:24.542037 20410 raft_consensus.cc:1261] T 7cd3b34b65f14fdda9728eda8d3672e5 P d406f7ab5f5a41ab80213fbb90a5ed3c [term 1 FOLLOWER]: Refusing update from remote peer 65a1a93520ed4f81a35dfaa5bb85c68f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:41:24.542711 20612 consensus_queue.cc:1035] T 7cd3b34b65f14fdda9728eda8d3672e5 P 65a1a93520ed4f81a35dfaa5bb85c68f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d406f7ab5f5a41ab80213fbb90a5ed3c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:41:24.542894 20541 raft_consensus.cc:1261] T 7cd3b34b65f14fdda9728eda8d3672e5 P 2328bfd725404468be67f4f08fec6590 [term 1 FOLLOWER]: Refusing update from remote peer 65a1a93520ed4f81a35dfaa5bb85c68f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:41:24.543444 20612 consensus_queue.cc:1035] T 7cd3b34b65f14fdda9728eda8d3672e5 P 65a1a93520ed4f81a35dfaa5bb85c68f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2328bfd725404468be67f4f08fec6590" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:41:24.551175 20619 mvcc.cc:205] Tried to move back new op lower bound from 7298580826277253120 to 7298580825992847360. Current Snapshot: MvccSnapshot[applied={T|T < 7298580826277253120}]
I20260619 14:41:24.558017 20136 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:41:24.558187 20136 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:41:24.560395 20136 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 76c136ccc7a94f7a99a9e790687d6197: Sending DeleteTablet for 3 replicas of tablet 7cd3b34b65f14fdda9728eda8d3672e5
I20260619 14:41:24.561088 20521 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cd3b34b65f14fdda9728eda8d3672e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:41:24 UTC) from {username='slave'} at 127.0.0.1:34526
I20260619 14:41:24.561228 20390 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cd3b34b65f14fdda9728eda8d3672e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:41:24 UTC) from {username='slave'} at 127.0.0.1:45406
I20260619 14:41:24.561553 20259 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cd3b34b65f14fdda9728eda8d3672e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:41:24 UTC) from {username='slave'} at 127.0.0.1:57028
I20260619 14:41:24.561759 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 20197
I20260619 14:41:24.561972 20629 tablet_replica.cc:330] T 7cd3b34b65f14fdda9728eda8d3672e5 P 2328bfd725404468be67f4f08fec6590: stopping tablet replica
I20260619 14:41:24.562372 20630 tablet_replica.cc:330] T 7cd3b34b65f14fdda9728eda8d3672e5 P d406f7ab5f5a41ab80213fbb90a5ed3c: stopping tablet replica
I20260619 14:41:24.562618 20630 raft_consensus.cc:2229] T 7cd3b34b65f14fdda9728eda8d3672e5 P d406f7ab5f5a41ab80213fbb90a5ed3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:41:24.562882 20630 raft_consensus.cc:2258] T 7cd3b34b65f14fdda9728eda8d3672e5 P d406f7ab5f5a41ab80213fbb90a5ed3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:41:24.563143 20629 raft_consensus.cc:2229] T 7cd3b34b65f14fdda9728eda8d3672e5 P 2328bfd725404468be67f4f08fec6590 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:41:24.563465 20629 raft_consensus.cc:2258] T 7cd3b34b65f14fdda9728eda8d3672e5 P 2328bfd725404468be67f4f08fec6590 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:41:24.564709 20630 ts_tablet_manager.cc:1905] T 7cd3b34b65f14fdda9728eda8d3672e5 P d406f7ab5f5a41ab80213fbb90a5ed3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:41:24.565152 20629 ts_tablet_manager.cc:1905] T 7cd3b34b65f14fdda9728eda8d3672e5 P 2328bfd725404468be67f4f08fec6590: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:41:24.567368 20630 ts_tablet_manager.cc:1918] T 7cd3b34b65f14fdda9728eda8d3672e5 P d406f7ab5f5a41ab80213fbb90a5ed3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:41:24.567464 20630 log.cc:1196] T 7cd3b34b65f14fdda9728eda8d3672e5 P d406f7ab5f5a41ab80213fbb90a5ed3c: Deleting WAL directory at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/ts-1/wal/wals/7cd3b34b65f14fdda9728eda8d3672e5
I20260619 14:41:24.567646 20629 ts_tablet_manager.cc:1918] T 7cd3b34b65f14fdda9728eda8d3672e5 P 2328bfd725404468be67f4f08fec6590: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:41:24.567752 20629 log.cc:1196] T 7cd3b34b65f14fdda9728eda8d3672e5 P 2328bfd725404468be67f4f08fec6590: Deleting WAL directory at /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781879837648036-392-0/minicluster-data/ts-2/wal/wals/7cd3b34b65f14fdda9728eda8d3672e5
I20260619 14:41:24.567770 20630 ts_tablet_manager.cc:1939] T 7cd3b34b65f14fdda9728eda8d3672e5 P d406f7ab5f5a41ab80213fbb90a5ed3c: Deleting consensus metadata
W20260619 14:41:24.567838 20122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46545 (error 108)
I20260619 14:41:24.568048 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 20328
I20260619 14:41:24.568068 20629 ts_tablet_manager.cc:1939] T 7cd3b34b65f14fdda9728eda8d3672e5 P 2328bfd725404468be67f4f08fec6590: Deleting consensus metadata
I20260619 14:41:24.569824 20124 catalog_manager.cc:4841] TS 2328bfd725404468be67f4f08fec6590 (127.0.98.3:40377): tablet 7cd3b34b65f14fdda9728eda8d3672e5 (table test-table [id=1f2d26112e7045e6bc5b24baee9bfbde]) successfully deleted
I20260619 14:41:24.570108 20599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:24.569869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46545 (local address 127.0.0.1:57054)
0619 14:41:24.569974 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:24.570040 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260619 14:41:24.570238 20122 catalog_manager.cc:4584] TS 65a1a93520ed4f81a35dfaa5bb85c68f (127.0.98.1:46545): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7cd3b34b65f14fdda9728eda8d3672e5: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46545: connect: Connection refused (error 111)
I20260619 14:41:24.572980 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 20459
I20260619 14:41:24.573451 20599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:24.573165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38739 (local address 127.0.0.1:45408)
0619 14:41:24.573325 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:24.573370 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:38739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260619 14:41:24.577512 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 20106
2026-06-19T14:41:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1252 ms)
[----------] 6 tests from EncryptionPolicyTest (9971 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781879837648036-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:41:24 dist-test-slave-32hj krb5kdc[20635](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:41:24 dist-test-slave-32hj krb5kdc[20635](info): set up 2 sockets
Jun 19 14:41:24 dist-test-slave-32hj krb5kdc[20635](info): commencing operation
krb5kdc: starting...
W20260619 14:41:25.536554 392 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781879837648036-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781879837648036-392-0/krb5kdc/test-user.keytab.
Jun 19 14:41:25 dist-test-slave-32hj krb5kdc[20635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:41:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:41:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781879837648036-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781879837648036-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:41:25.647581 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781879837648036-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781879837648036-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781879837648036-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45379
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781879837648036-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45379
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:41:25.753383 20648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:41:25.753645 20648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:41:25.753685 20648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:41:25.757612 20648 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:41:25.757673 20648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:41:25.757691 20648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:41:25.757707 20648 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:41:25.757726 20648 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:41:25.763049 20648 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781879837648036-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781879837648036-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45379
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781879837648036-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781879837648036-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:41:25.764035 20648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:41:25.764992 20648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:41:25.773145 20653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:25.773222 20648 server_base.cc:1034] running on GCE node
W20260619 14:41:25.773133 20656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:25.773180 20654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:25.773960 20648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:41:25.775005 20648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:41:25.776186 20648 hybrid_clock.cc:648] HybridClock initialized: now 1781880085776166 us; error 37 us; skew 500 ppm
Jun 19 14:41:25 dist-test-slave-32hj krb5kdc[20635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880085, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:41:25.778997 20648 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:41:25.779911 20648 webserver.cc:460] Webserver started at http://127.0.0.1:45537/ using document root <none> and password file <none>
I20260619 14:41:25.780442 20648 fs_manager.cc:359] Metadata directory not provided
I20260619 14:41:25.780503 20648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:41:25.780706 20648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:41:25.782793 20648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781879837648036-392-0/minicluster-data/master-0/data/instance:
uuid: "0175e6ca279c43fdbd3c8ca2bfcbdc7a"
format_stamp: "Formatted at 2026-06-19 14:41:25 on dist-test-slave-32hj"
server_key: "67c1622f8467541fdfe0395a839ccb71"
server_key_iv: "dd74c3704e249cc78e84f6e45e404e24"
server_key_version: "encryptionkey@0"
I20260619 14:41:25.783313 20648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781879837648036-392-0/minicluster-data/master-0/wal/instance:
uuid: "0175e6ca279c43fdbd3c8ca2bfcbdc7a"
format_stamp: "Formatted at 2026-06-19 14:41:25 on dist-test-slave-32hj"
server_key: "67c1622f8467541fdfe0395a839ccb71"
server_key_iv: "dd74c3704e249cc78e84f6e45e404e24"
server_key_version: "encryptionkey@0"
I20260619 14:41:25.786923 20648 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:41:25.789256 20663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:25.790496 20648 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:41:25.790637 20648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781879837648036-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781879837648036-392-0/minicluster-data/master-0/wal
uuid: "0175e6ca279c43fdbd3c8ca2bfcbdc7a"
format_stamp: "Formatted at 2026-06-19 14:41:25 on dist-test-slave-32hj"
server_key: "67c1622f8467541fdfe0395a839ccb71"
server_key_iv: "dd74c3704e249cc78e84f6e45e404e24"
server_key_version: "encryptionkey@0"
I20260619 14:41:25.790746 20648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781879837648036-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781879837648036-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781879837648036-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:41:25.808569 20648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:41:25.811914 20648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:41:25.812104 20648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:41:25.819154 20648 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45379
I20260619 14:41:25.819217 20715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45379 every 8 connection(s)
I20260619 14:41:25.820128 20648 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:41:25.822877 20716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:41:25.823138 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 20648
I20260619 14:41:25.823263 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781879837648036-392-0/minicluster-data/master-0/wal/instance
I20260619 14:41:25.823575 392 external_mini_cluster.cc:1428] Setting key 4deb4805ae4d7e35f5ca1370a9b6e15b
I20260619 14:41:25.829836 20716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0175e6ca279c43fdbd3c8ca2bfcbdc7a: Bootstrap starting.
I20260619 14:41:25.832515 20716 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0175e6ca279c43fdbd3c8ca2bfcbdc7a: Neither blocks nor log segments found. Creating new log.
Jun 19 14:41:25 dist-test-slave-32hj krb5kdc[20635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 14:41:25.833283 20716 log.cc:824] T 00000000000000000000000000000000 P 0175e6ca279c43fdbd3c8ca2bfcbdc7a: Log is configured to *not* fsync() on all Append() calls
I20260619 14:41:25.835083 20716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0175e6ca279c43fdbd3c8ca2bfcbdc7a: No bootstrap required, opened a new log
I20260619 14:41:25.837697 20719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:25.825084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46369 (local address 127.0.0.1:45379)
0619 14:41:25.825519 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:25.825534 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:25.825569 (+ 35us) server_negotiation.cc:407] Connection header received
0619 14:41:25.826262 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:25.826279 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:25.826621 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:25.826915 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:25.827834 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:25.828864 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:25.829995 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:25.830552 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:25.830669 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:25.833369 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:25.833375 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:25.833393 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:41:25.833425 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:25.834944 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:25.835451 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:25.835456 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:25.835460 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:25.835566 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:25.835883 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:25.835886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:25.835887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:25.836274 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:25.836465 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:25.836787 (+ 322us) server_negotiation.cc:299] Negotiation successful
0619 14:41:25.836992 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":139,"threads_started":1}
I20260619 14:41:25.838851 20716 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0175e6ca279c43fdbd3c8ca2bfcbdc7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0175e6ca279c43fdbd3c8ca2bfcbdc7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45379 } }
I20260619 14:41:25.839067 20716 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0175e6ca279c43fdbd3c8ca2bfcbdc7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:41:25.839128 20716 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0175e6ca279c43fdbd3c8ca2bfcbdc7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0175e6ca279c43fdbd3c8ca2bfcbdc7a, State: Initialized, Role: FOLLOWER
I20260619 14:41:25.839529 20716 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0175e6ca279c43fdbd3c8ca2bfcbdc7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0175e6ca279c43fdbd3c8ca2bfcbdc7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45379 } }
I20260619 14:41:25.839684 20716 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0175e6ca279c43fdbd3c8ca2bfcbdc7a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:41:25.839749 20716 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0175e6ca279c43fdbd3c8ca2bfcbdc7a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:41:25.839843 20716 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0175e6ca279c43fdbd3c8ca2bfcbdc7a [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:41:25.840875 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:41:25.840874 20716 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0175e6ca279c43fdbd3c8ca2bfcbdc7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0175e6ca279c43fdbd3c8ca2bfcbdc7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45379 } }
I20260619 14:41:25.841214 20716 leader_election.cc:302] T 00000000000000000000000000000000 P 0175e6ca279c43fdbd3c8ca2bfcbdc7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0175e6ca279c43fdbd3c8ca2bfcbdc7a; no voters:
I20260619 14:41:25.841545 20716 leader_election.cc:288] T 00000000000000000000000000000000 P 0175e6ca279c43fdbd3c8ca2bfcbdc7a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:41:25.841637 20721 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0175e6ca279c43fdbd3c8ca2bfcbdc7a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:41:25.841849 20721 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0175e6ca279c43fdbd3c8ca2bfcbdc7a [term 1 LEADER]: Becoming Leader. State: Replica: 0175e6ca279c43fdbd3c8ca2bfcbdc7a, State: Running, Role: LEADER
I20260619 14:41:25.842149 20721 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0175e6ca279c43fdbd3c8ca2bfcbdc7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0175e6ca279c43fdbd3c8ca2bfcbdc7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45379 } }
I20260619 14:41:25.842870 20716 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0175e6ca279c43fdbd3c8ca2bfcbdc7a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:41:25.843660 20723 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0175e6ca279c43fdbd3c8ca2bfcbdc7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0175e6ca279c43fdbd3c8ca2bfcbdc7a. Latest consensus state: current_term: 1 leader_uuid: "0175e6ca279c43fdbd3c8ca2bfcbdc7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0175e6ca279c43fdbd3c8ca2bfcbdc7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45379 } } }
I20260619 14:41:25.843978 20722 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0175e6ca279c43fdbd3c8ca2bfcbdc7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0175e6ca279c43fdbd3c8ca2bfcbdc7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0175e6ca279c43fdbd3c8ca2bfcbdc7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45379 } } }
I20260619 14:41:25.844089 20722 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0175e6ca279c43fdbd3c8ca2bfcbdc7a [sys.catalog]: This master's current role is: LEADER
I20260619 14:41:25.844388 20723 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0175e6ca279c43fdbd3c8ca2bfcbdc7a [sys.catalog]: This master's current role is: LEADER
I20260619 14:41:25.844506 20735 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:41:25.847666 20735 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:41:25.851805 20719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:25.843085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46075 (local address 127.0.0.1:45379)
0619 14:41:25.843189 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:25.843193 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:25.843491 (+ 298us) server_negotiation.cc:407] Connection header received
0619 14:41:25.843619 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:25.843622 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:25.843671 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:25.843766 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:25.844537 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:25.845277 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:25.846342 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:25.846648 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:25.846758 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:25.848374 (+ 1616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:25.848378 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:25.848381 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:41:25.848406 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:25.850061 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:25.850535 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:25.850538 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:25.850540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:25.850612 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:25.850903 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:25.850906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:25.850908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:25.851068 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:25.851193 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:25.851448 (+ 255us) server_negotiation.cc:299] Negotiation successful
0619 14:41:25.851534 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 14:41:25.854323 20735 catalog_manager.cc:1347] Generated new cluster ID: 119be8df104c4c8c9daaac15881de1e2
I20260619 14:41:25.854389 20735 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:41:25.864032 20735 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:41:25.865301 20735 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:41:25.882655 20735 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0175e6ca279c43fdbd3c8ca2bfcbdc7a: Generated new TSK 0
I20260619 14:41:25.883530 20735 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:41:25.888145 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 20648
W20260619 14:41:25.892616 20633 connection.cc:535] client connection to 127.0.0.1:45379 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45379): Connection reset by peer (error 104)
2026-06-19T14:41:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1309 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781879837648036-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:41:25 dist-test-slave-32hj krb5kdc[20747](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:41:25 dist-test-slave-32hj krb5kdc[20747](info): set up 2 sockets
Jun 19 14:41:25 dist-test-slave-32hj krb5kdc[20747](info): commencing operation
krb5kdc: starting...
W20260619 14:41:26.840603 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781879837648036-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781879837648036-392-0/krb5kdc/test-user.keytab.
Jun 19 14:41:26 dist-test-slave-32hj krb5kdc[20747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:41:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:41:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781879837648036-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781879837648036-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:41:26.957721 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781879837648036-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781879837648036-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781879837648036-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35411
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781879837648036-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35411
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:41:27.067358 20760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:41:27.067656 20760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:41:27.067732 20760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:41:27.071813 20760 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:41:27.071897 20760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:41:27.071926 20760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:41:27.071946 20760 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:41:27.071967 20760 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:41:27.077409 20760 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781879837648036-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781879837648036-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35411
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781879837648036-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781879837648036-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:41:27.078507 20760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:41:27.079517 20760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:41:27.087881 20768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:27.087898 20765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:27.087944 20760 server_base.cc:1034] running on GCE node
W20260619 14:41:27.087898 20766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:27.088702 20760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:41:27.089888 20760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:41:27.091074 20760 hybrid_clock.cc:648] HybridClock initialized: now 1781880087091055 us; error 34 us; skew 500 ppm
Jun 19 14:41:27 dist-test-slave-32hj krb5kdc[20747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880087, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:41:27.093907 20760 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:41:27.094803 20760 webserver.cc:460] Webserver started at http://127.0.0.1:40227/ using document root <none> and password file <none>
I20260619 14:41:27.095329 20760 fs_manager.cc:359] Metadata directory not provided
I20260619 14:41:27.095384 20760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:41:27.095559 20760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:41:27.097671 20760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781879837648036-392-0/minicluster-data/master-0/data/instance:
uuid: "0641eb9233a040929275402c93e4d473"
format_stamp: "Formatted at 2026-06-19 14:41:27 on dist-test-slave-32hj"
server_key: "0385f055228941da23826bb918cd2836"
server_key_iv: "61238e1a3e24d3b866ab7dd3d4405501"
server_key_version: "encryptionkey@0"
I20260619 14:41:27.098192 20760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781879837648036-392-0/minicluster-data/master-0/wal/instance:
uuid: "0641eb9233a040929275402c93e4d473"
format_stamp: "Formatted at 2026-06-19 14:41:27 on dist-test-slave-32hj"
server_key: "0385f055228941da23826bb918cd2836"
server_key_iv: "61238e1a3e24d3b866ab7dd3d4405501"
server_key_version: "encryptionkey@0"
I20260619 14:41:27.101910 20760 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260619 14:41:27.104380 20775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:27.105669 20760 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:41:27.105777 20760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781879837648036-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781879837648036-392-0/minicluster-data/master-0/wal
uuid: "0641eb9233a040929275402c93e4d473"
format_stamp: "Formatted at 2026-06-19 14:41:27 on dist-test-slave-32hj"
server_key: "0385f055228941da23826bb918cd2836"
server_key_iv: "61238e1a3e24d3b866ab7dd3d4405501"
server_key_version: "encryptionkey@0"
I20260619 14:41:27.105894 20760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781879837648036-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781879837648036-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781879837648036-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:41:27.138091 20760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:41:27.141805 20760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:41:27.142002 20760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:41:27.149169 20760 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35411
I20260619 14:41:27.149253 20827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35411 every 8 connection(s)
I20260619 14:41:27.150157 20760 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:41:27.152953 20828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:41:27.154286 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 20760
I20260619 14:41:27.154412 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781879837648036-392-0/minicluster-data/master-0/wal/instance
I20260619 14:41:27.154680 392 external_mini_cluster.cc:1428] Setting key 29afda7f08a36bf009a8419332e7021c
I20260619 14:41:27.159575 20828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0641eb9233a040929275402c93e4d473: Bootstrap starting.
I20260619 14:41:27.162312 20828 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0641eb9233a040929275402c93e4d473: Neither blocks nor log segments found. Creating new log.
I20260619 14:41:27.163126 20828 log.cc:824] T 00000000000000000000000000000000 P 0641eb9233a040929275402c93e4d473: Log is configured to *not* fsync() on all Append() calls
I20260619 14:41:27.165234 20831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:27.156080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37973 (local address 127.0.0.1:35411)
0619 14:41:27.156543 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:27.156561 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:27.156606 (+ 45us) server_negotiation.cc:407] Connection header received
0619 14:41:27.157509 (+ 903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:27.157534 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:27.157983 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:27.158397 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:27.158813 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:27.160263 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:27.161278 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:27.161938 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:27.162114 (+ 176us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:27.162926 (+ 812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:27.162933 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:27.162949 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:41:27.163463 (+ 514us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:27.163532 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:41:27.163564 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:27.163682 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:27.164021 (+ 339us) server_negotiation.cc:299] Negotiation successful
0619 14:41:27.164181 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":153,"threads_started":1}
I20260619 14:41:27.165378 20828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0641eb9233a040929275402c93e4d473: No bootstrap required, opened a new log
I20260619 14:41:27.168423 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:41:27.169528 20828 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0641eb9233a040929275402c93e4d473 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0641eb9233a040929275402c93e4d473" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35411 } }
I20260619 14:41:27.169814 20828 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0641eb9233a040929275402c93e4d473 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:41:27.169886 20828 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0641eb9233a040929275402c93e4d473 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0641eb9233a040929275402c93e4d473, State: Initialized, Role: FOLLOWER
I20260619 14:41:27.170347 20828 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0641eb9233a040929275402c93e4d473 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0641eb9233a040929275402c93e4d473" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35411 } }
I20260619 14:41:27.170511 20828 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0641eb9233a040929275402c93e4d473 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:41:27.170622 20828 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0641eb9233a040929275402c93e4d473 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:41:27.170732 20828 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0641eb9233a040929275402c93e4d473 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:41:27.171926 20828 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0641eb9233a040929275402c93e4d473 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0641eb9233a040929275402c93e4d473" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35411 } }
I20260619 14:41:27.172307 20828 leader_election.cc:302] T 00000000000000000000000000000000 P 0641eb9233a040929275402c93e4d473 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0641eb9233a040929275402c93e4d473; no voters:
I20260619 14:41:27.172595 20828 leader_election.cc:288] T 00000000000000000000000000000000 P 0641eb9233a040929275402c93e4d473 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:41:27.172703 20838 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0641eb9233a040929275402c93e4d473 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:41:27.172919 20838 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0641eb9233a040929275402c93e4d473 [term 1 LEADER]: Becoming Leader. State: Replica: 0641eb9233a040929275402c93e4d473, State: Running, Role: LEADER
I20260619 14:41:27.173323 20838 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0641eb9233a040929275402c93e4d473 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0641eb9233a040929275402c93e4d473" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35411 } }
I20260619 14:41:27.174021 20828 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0641eb9233a040929275402c93e4d473 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:41:27.174988 20840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0641eb9233a040929275402c93e4d473 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0641eb9233a040929275402c93e4d473. Latest consensus state: current_term: 1 leader_uuid: "0641eb9233a040929275402c93e4d473" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0641eb9233a040929275402c93e4d473" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35411 } } }
I20260619 14:41:27.175206 20831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:27.170478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53331 (local address 127.0.0.1:35411)
0619 14:41:27.170596 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:27.170601 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:27.170809 (+ 208us) server_negotiation.cc:407] Connection header received
0619 14:41:27.170948 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:27.170951 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:27.171002 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:27.171082 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:27.171359 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:27.172099 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:27.173153 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:27.173480 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:27.173587 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:27.174437 (+ 850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:27.174441 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:27.174444 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:41:27.174455 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:27.174482 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:41:27.174502 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:27.174585 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:27.175039 (+ 454us) server_negotiation.cc:299] Negotiation successful
0619 14:41:27.175077 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 14:41:27.175367 20840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0641eb9233a040929275402c93e4d473 [sys.catalog]: This master's current role is: LEADER
I20260619 14:41:27.175261 20839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0641eb9233a040929275402c93e4d473 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0641eb9233a040929275402c93e4d473" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0641eb9233a040929275402c93e4d473" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35411 } } }
I20260619 14:41:27.175625 20839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0641eb9233a040929275402c93e4d473 [sys.catalog]: This master's current role is: LEADER
I20260619 14:41:27.176254 20848 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:41:27.178922 20848 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:41:27.183981 20848 catalog_manager.cc:1347] Generated new cluster ID: c7291d91b3c0429c86716bd0b8bba541
I20260619 14:41:27.184062 20848 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:41:27.211305 20848 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:41:27.212265 20848 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:41:27.220234 20848 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0641eb9233a040929275402c93e4d473: Generated new TSK 0
I20260619 14:41:27.221027 20848 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:41:27.231432 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 20760
W20260619 14:41:27.235999 20745 connection.cc:535] client connection to 127.0.0.1:35411 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35411): Connection reset by peer (error 104)
2026-06-19T14:41:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1343 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781879837648036-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:41:27 dist-test-slave-32hj krb5kdc[20859](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:41:27 dist-test-slave-32hj krb5kdc[20859](info): set up 2 sockets
Jun 19 14:41:27 dist-test-slave-32hj krb5kdc[20859](info): commencing operation
krb5kdc: starting...
W20260619 14:41:28.188606 392 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781879837648036-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781879837648036-392-0/krb5kdc/test-user.keytab.
Jun 19 14:41:28 dist-test-slave-32hj krb5kdc[20859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:41:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:41:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781879837648036-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781879837648036-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:41:28.306118 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781879837648036-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781879837648036-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781879837648036-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32851
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781879837648036-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32851
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:41:28.413296 20872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:41:28.413570 20872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:41:28.413611 20872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:41:28.417645 20872 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:41:28.417714 20872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:41:28.417733 20872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:41:28.417748 20872 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:41:28.417762 20872 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:41:28.423156 20872 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781879837648036-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781879837648036-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32851
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781879837648036-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781879837648036-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:41:28.424242 20872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:41:28.425283 20872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:41:28.433703 20877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:28.433866 20872 server_base.cc:1034] running on GCE node
W20260619 14:41:28.433909 20878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:28.433725 20880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:28.434561 20872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:41:28.435772 20872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:41:28.436920 20872 hybrid_clock.cc:648] HybridClock initialized: now 1781880088436902 us; error 32 us; skew 500 ppm
Jun 19 14:41:28 dist-test-slave-32hj krb5kdc[20859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880088, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:41:28.443859 20872 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:41:28.444829 20872 webserver.cc:460] Webserver started at http://127.0.0.1:44283/ using document root <none> and password file <none>
I20260619 14:41:28.445385 20872 fs_manager.cc:359] Metadata directory not provided
I20260619 14:41:28.445453 20872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:41:28.445652 20872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:41:28.447692 20872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781879837648036-392-0/minicluster-data/master-0/data/instance:
uuid: "027e8814d3b641f2a8df4b6e505c63bc"
format_stamp: "Formatted at 2026-06-19 14:41:28 on dist-test-slave-32hj"
server_key: "2bb1752135f8a89c48fa9053b8fa3cad"
server_key_iv: "48f550f09bb1dd7f5352bb605a237d9f"
server_key_version: "encryptionkey@0"
I20260619 14:41:28.448244 20872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781879837648036-392-0/minicluster-data/master-0/wal/instance:
uuid: "027e8814d3b641f2a8df4b6e505c63bc"
format_stamp: "Formatted at 2026-06-19 14:41:28 on dist-test-slave-32hj"
server_key: "2bb1752135f8a89c48fa9053b8fa3cad"
server_key_iv: "48f550f09bb1dd7f5352bb605a237d9f"
server_key_version: "encryptionkey@0"
I20260619 14:41:28.452102 20872 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260619 14:41:28.454730 20887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:28.456053 20872 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 14:41:28.456174 20872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781879837648036-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781879837648036-392-0/minicluster-data/master-0/wal
uuid: "027e8814d3b641f2a8df4b6e505c63bc"
format_stamp: "Formatted at 2026-06-19 14:41:28 on dist-test-slave-32hj"
server_key: "2bb1752135f8a89c48fa9053b8fa3cad"
server_key_iv: "48f550f09bb1dd7f5352bb605a237d9f"
server_key_version: "encryptionkey@0"
I20260619 14:41:28.456282 20872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781879837648036-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781879837648036-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781879837648036-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:41:28.460889 20872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:41:28.463905 20872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:41:28.464100 20872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:41:28.470640 20872 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32851
I20260619 14:41:28.470674 20939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32851 every 8 connection(s)
I20260619 14:41:28.471570 20872 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:41:28.472100 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 20872
I20260619 14:41:28.472242 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781879837648036-392-0/minicluster-data/master-0/wal/instance
I20260619 14:41:28.472484 392 external_mini_cluster.cc:1428] Setting key 019b5f0b1fd282b662d0ba7992d01687
I20260619 14:41:28.474447 20940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:41:28.477717 20943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:28.473759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54881 (local address 127.0.0.1:32851)
0619 14:41:28.474174 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:28.474194 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:28.474383 (+ 189us) server_negotiation.cc:407] Connection header received
0619 14:41:28.475107 (+ 724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:28.475125 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:28.475421 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:28.475736 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:28.476528 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:28.476532 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:28.476544 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:41:28.476554 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:28.476612 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:41:28.476631 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:28.476786 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:28.476857 (+ 71us) server_negotiation.cc:299] Negotiation successful
0619 14:41:28.477011 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":111,"threads_started":1}
I20260619 14:41:28.480607 20940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 027e8814d3b641f2a8df4b6e505c63bc: Bootstrap starting.
I20260619 14:41:28.480791 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:41:28.483070 20940 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 027e8814d3b641f2a8df4b6e505c63bc: Neither blocks nor log segments found. Creating new log.
I20260619 14:41:28.483906 20940 log.cc:824] T 00000000000000000000000000000000 P 027e8814d3b641f2a8df4b6e505c63bc: Log is configured to *not* fsync() on all Append() calls
I20260619 14:41:28.484364 20943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:28.482764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57511 (local address 127.0.0.1:32851)
0619 14:41:28.482873 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:28.482877 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:28.483047 (+ 170us) server_negotiation.cc:407] Connection header received
0619 14:41:28.483172 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:28.483175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:28.483211 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:28.483274 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:28.483959 (+ 685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:28.483963 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:28.483966 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:41:28.483977 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:28.484004 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:41:28.484015 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:28.484094 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:28.484214 (+ 120us) server_negotiation.cc:299] Negotiation successful
0619 14:41:28.484249 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 14:41:28.486235 20940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 027e8814d3b641f2a8df4b6e505c63bc: No bootstrap required, opened a new log
I20260619 14:41:28.489704 20940 raft_consensus.cc:348] T 00000000000000000000000000000000 P 027e8814d3b641f2a8df4b6e505c63bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "027e8814d3b641f2a8df4b6e505c63bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32851 } }
I20260619 14:41:28.489933 20940 raft_consensus.cc:374] T 00000000000000000000000000000000 P 027e8814d3b641f2a8df4b6e505c63bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:41:28.489987 20940 raft_consensus.cc:729] T 00000000000000000000000000000000 P 027e8814d3b641f2a8df4b6e505c63bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 027e8814d3b641f2a8df4b6e505c63bc, State: Initialized, Role: FOLLOWER
I20260619 14:41:28.490377 20940 consensus_queue.cc:260] T 00000000000000000000000000000000 P 027e8814d3b641f2a8df4b6e505c63bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "027e8814d3b641f2a8df4b6e505c63bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32851 } }
I20260619 14:41:28.490506 20940 raft_consensus.cc:388] T 00000000000000000000000000000000 P 027e8814d3b641f2a8df4b6e505c63bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:41:28.490568 20940 raft_consensus.cc:482] T 00000000000000000000000000000000 P 027e8814d3b641f2a8df4b6e505c63bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:41:28.490664 20940 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 027e8814d3b641f2a8df4b6e505c63bc [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:41:28.491637 20940 raft_consensus.cc:504] T 00000000000000000000000000000000 P 027e8814d3b641f2a8df4b6e505c63bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "027e8814d3b641f2a8df4b6e505c63bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32851 } }
I20260619 14:41:28.491941 20940 leader_election.cc:302] T 00000000000000000000000000000000 P 027e8814d3b641f2a8df4b6e505c63bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 027e8814d3b641f2a8df4b6e505c63bc; no voters:
I20260619 14:41:28.492215 20940 leader_election.cc:288] T 00000000000000000000000000000000 P 027e8814d3b641f2a8df4b6e505c63bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:41:28.492311 20950 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 027e8814d3b641f2a8df4b6e505c63bc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:41:28.492529 20950 raft_consensus.cc:686] T 00000000000000000000000000000000 P 027e8814d3b641f2a8df4b6e505c63bc [term 1 LEADER]: Becoming Leader. State: Replica: 027e8814d3b641f2a8df4b6e505c63bc, State: Running, Role: LEADER
I20260619 14:41:28.492885 20950 consensus_queue.cc:237] T 00000000000000000000000000000000 P 027e8814d3b641f2a8df4b6e505c63bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "027e8814d3b641f2a8df4b6e505c63bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32851 } }
I20260619 14:41:28.493400 20940 sys_catalog.cc:564] T 00000000000000000000000000000000 P 027e8814d3b641f2a8df4b6e505c63bc [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:41:28.494601 20952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 027e8814d3b641f2a8df4b6e505c63bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 027e8814d3b641f2a8df4b6e505c63bc. Latest consensus state: current_term: 1 leader_uuid: "027e8814d3b641f2a8df4b6e505c63bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "027e8814d3b641f2a8df4b6e505c63bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32851 } } }
I20260619 14:41:28.494607 20951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 027e8814d3b641f2a8df4b6e505c63bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "027e8814d3b641f2a8df4b6e505c63bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "027e8814d3b641f2a8df4b6e505c63bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32851 } } }
I20260619 14:41:28.494807 20951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 027e8814d3b641f2a8df4b6e505c63bc [sys.catalog]: This master's current role is: LEADER
I20260619 14:41:28.494800 20952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 027e8814d3b641f2a8df4b6e505c63bc [sys.catalog]: This master's current role is: LEADER
W20260619 14:41:28.497872 20965 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 027e8814d3b641f2a8df4b6e505c63bc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:41:28.497944 20965 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:41:28.498030 20961 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:41:28.498890 20961 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:41:28.504447 20961 catalog_manager.cc:1347] Generated new cluster ID: 38b196a33679400fbb3f90304770980c
I20260619 14:41:28.504516 20961 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:41:28.520555 20961 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:41:28.520936 20961 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:41:28.524580 20961 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 027e8814d3b641f2a8df4b6e505c63bc: Generated new TSK 0
I20260619 14:41:28.525399 20961 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:41:28.528258 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 20872
2026-06-19T14:41:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1296 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781879837648036-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:41:28 dist-test-slave-32hj krb5kdc[20971](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:41:28 dist-test-slave-32hj krb5kdc[20971](info): set up 2 sockets
Jun 19 14:41:28 dist-test-slave-32hj krb5kdc[20971](info): commencing operation
krb5kdc: starting...
W20260619 14:41:29.480578 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781879837648036-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781879837648036-392-0/krb5kdc/test-user.keytab.
Jun 19 14:41:29 dist-test-slave-32hj krb5kdc[20971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:41:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:41:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781879837648036-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781879837648036-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:41:29.593897 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781879837648036-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781879837648036-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781879837648036-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46141
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781879837648036-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46141
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:41:29.701225 20984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:41:29.701493 20984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:41:29.701542 20984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:41:29.705542 20984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:41:29.705613 20984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:41:29.705633 20984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:41:29.705649 20984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:41:29.705663 20984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:41:29.711079 20984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781879837648036-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781879837648036-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781879837648036-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781879837648036-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:41:29.712138 20984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:41:29.713182 20984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:41:29.721486 20992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:29.721479 20990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:29.721482 20989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:29.722066 20984 server_base.cc:1034] running on GCE node
I20260619 14:41:29.722621 20984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:41:29.723698 20984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:41:29.724808 20984 hybrid_clock.cc:648] HybridClock initialized: now 1781880089724788 us; error 30 us; skew 500 ppm
Jun 19 14:41:29 dist-test-slave-32hj krb5kdc[20971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880089, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:41:29.727541 20984 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:41:29.728463 20984 webserver.cc:460] Webserver started at http://127.0.0.1:42063/ using document root <none> and password file <none>
I20260619 14:41:29.729046 20984 fs_manager.cc:359] Metadata directory not provided
I20260619 14:41:29.729116 20984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:41:29.729314 20984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:41:29.731346 20984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781879837648036-392-0/minicluster-data/master-0/data/instance:
uuid: "b780cfa2defe47ac9cbd8e6cccae3e07"
format_stamp: "Formatted at 2026-06-19 14:41:29 on dist-test-slave-32hj"
server_key: "98ddf0d3c339a0d32e4e71bfb6310d74"
server_key_iv: "2c920362760fec2e4df15283f3aa7058"
server_key_version: "encryptionkey@0"
I20260619 14:41:29.731877 20984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781879837648036-392-0/minicluster-data/master-0/wal/instance:
uuid: "b780cfa2defe47ac9cbd8e6cccae3e07"
format_stamp: "Formatted at 2026-06-19 14:41:29 on dist-test-slave-32hj"
server_key: "98ddf0d3c339a0d32e4e71bfb6310d74"
server_key_iv: "2c920362760fec2e4df15283f3aa7058"
server_key_version: "encryptionkey@0"
I20260619 14:41:29.735497 20984 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 14:41:29.737875 20999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:29.739113 20984 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:41:29.739218 20984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781879837648036-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781879837648036-392-0/minicluster-data/master-0/wal
uuid: "b780cfa2defe47ac9cbd8e6cccae3e07"
format_stamp: "Formatted at 2026-06-19 14:41:29 on dist-test-slave-32hj"
server_key: "98ddf0d3c339a0d32e4e71bfb6310d74"
server_key_iv: "2c920362760fec2e4df15283f3aa7058"
server_key_version: "encryptionkey@0"
I20260619 14:41:29.739328 20984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781879837648036-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781879837648036-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781879837648036-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:41:29.751763 20984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:41:29.754987 20984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:41:29.755148 20984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:41:29.761678 20984 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46141
I20260619 14:41:29.761698 21051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46141 every 8 connection(s)
I20260619 14:41:29.762581 20984 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:41:29.765331 21052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:41:29.769868 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 20984
I20260619 14:41:29.769989 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781879837648036-392-0/minicluster-data/master-0/wal/instance
I20260619 14:41:29.770205 392 external_mini_cluster.cc:1428] Setting key b2f7daf9e9138af904645b959c1b275e
I20260619 14:41:29.770758 21052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b780cfa2defe47ac9cbd8e6cccae3e07: Bootstrap starting.
I20260619 14:41:29.772991 21052 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b780cfa2defe47ac9cbd8e6cccae3e07: Neither blocks nor log segments found. Creating new log.
I20260619 14:41:29.773857 21052 log.cc:824] T 00000000000000000000000000000000 P b780cfa2defe47ac9cbd8e6cccae3e07: Log is configured to *not* fsync() on all Append() calls
I20260619 14:41:29.775847 21052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b780cfa2defe47ac9cbd8e6cccae3e07: No bootstrap required, opened a new log
Jun 19 14:41:29 dist-test-slave-32hj krb5kdc[20971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 14:41:29.780012 21052 raft_consensus.cc:348] T 00000000000000000000000000000000 P b780cfa2defe47ac9cbd8e6cccae3e07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b780cfa2defe47ac9cbd8e6cccae3e07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46141 } }
I20260619 14:41:29.780306 21052 raft_consensus.cc:374] T 00000000000000000000000000000000 P b780cfa2defe47ac9cbd8e6cccae3e07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:41:29.780373 21052 raft_consensus.cc:729] T 00000000000000000000000000000000 P b780cfa2defe47ac9cbd8e6cccae3e07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b780cfa2defe47ac9cbd8e6cccae3e07, State: Initialized, Role: FOLLOWER
I20260619 14:41:29.780893 21052 consensus_queue.cc:260] T 00000000000000000000000000000000 P b780cfa2defe47ac9cbd8e6cccae3e07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b780cfa2defe47ac9cbd8e6cccae3e07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46141 } }
I20260619 14:41:29.781023 21052 raft_consensus.cc:388] T 00000000000000000000000000000000 P b780cfa2defe47ac9cbd8e6cccae3e07 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:41:29.781087 21052 raft_consensus.cc:482] T 00000000000000000000000000000000 P b780cfa2defe47ac9cbd8e6cccae3e07 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:41:29.781189 21052 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b780cfa2defe47ac9cbd8e6cccae3e07 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:41:29.782188 21052 raft_consensus.cc:504] T 00000000000000000000000000000000 P b780cfa2defe47ac9cbd8e6cccae3e07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b780cfa2defe47ac9cbd8e6cccae3e07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46141 } }
I20260619 14:41:29.782546 21052 leader_election.cc:302] T 00000000000000000000000000000000 P b780cfa2defe47ac9cbd8e6cccae3e07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b780cfa2defe47ac9cbd8e6cccae3e07; no voters:
I20260619 14:41:29.782866 21052 leader_election.cc:288] T 00000000000000000000000000000000 P b780cfa2defe47ac9cbd8e6cccae3e07 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:41:29.782948 21057 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b780cfa2defe47ac9cbd8e6cccae3e07 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:41:29.783113 21057 raft_consensus.cc:686] T 00000000000000000000000000000000 P b780cfa2defe47ac9cbd8e6cccae3e07 [term 1 LEADER]: Becoming Leader. State: Replica: b780cfa2defe47ac9cbd8e6cccae3e07, State: Running, Role: LEADER
I20260619 14:41:29.783412 21057 consensus_queue.cc:237] T 00000000000000000000000000000000 P b780cfa2defe47ac9cbd8e6cccae3e07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b780cfa2defe47ac9cbd8e6cccae3e07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46141 } }
I20260619 14:41:29.784183 21052 sys_catalog.cc:564] T 00000000000000000000000000000000 P b780cfa2defe47ac9cbd8e6cccae3e07 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:41:29.785219 21058 sys_catalog.cc:455] T 00000000000000000000000000000000 P b780cfa2defe47ac9cbd8e6cccae3e07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b780cfa2defe47ac9cbd8e6cccae3e07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b780cfa2defe47ac9cbd8e6cccae3e07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46141 } } }
I20260619 14:41:29.785221 21059 sys_catalog.cc:455] T 00000000000000000000000000000000 P b780cfa2defe47ac9cbd8e6cccae3e07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b780cfa2defe47ac9cbd8e6cccae3e07. Latest consensus state: current_term: 1 leader_uuid: "b780cfa2defe47ac9cbd8e6cccae3e07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b780cfa2defe47ac9cbd8e6cccae3e07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46141 } } }
I20260619 14:41:29.785377 21058 sys_catalog.cc:458] T 00000000000000000000000000000000 P b780cfa2defe47ac9cbd8e6cccae3e07 [sys.catalog]: This master's current role is: LEADER
I20260619 14:41:29.785380 21059 sys_catalog.cc:458] T 00000000000000000000000000000000 P b780cfa2defe47ac9cbd8e6cccae3e07 [sys.catalog]: This master's current role is: LEADER
I20260619 14:41:29.785223 21055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:29.771576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46703 (local address 127.0.0.1:46141)
0619 14:41:29.771972 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:29.771988 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:29.772023 (+ 35us) server_negotiation.cc:407] Connection header received
0619 14:41:29.772646 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:29.772662 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:29.773040 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:29.773344 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:29.774226 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:29.775180 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:29.776303 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:29.776813 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:29.776937 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:29.779653 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:29.779659 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:29.779672 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:41:29.779699 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:29.781600 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:29.782088 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:29.782093 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:29.782097 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:29.782233 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:29.782532 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:29.782534 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:29.782535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:29.783055 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:29.783836 (+ 781us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:29.783957 (+ 121us) server_negotiation.cc:299] Negotiation successful
0619 14:41:29.784224 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":98,"threads_started":1}
I20260619 14:41:29.785921 21066 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:41:29.788466 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:41:29.789134 21066 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:41:29.797626 21066 catalog_manager.cc:1347] Generated new cluster ID: ea891b94d9364f08bbe9efc831047c6f
I20260619 14:41:29.797729 21066 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:41:29.803371 21055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:29.795778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47909 (local address 127.0.0.1:46141)
0619 14:41:29.795895 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:29.795900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:29.796168 (+ 268us) server_negotiation.cc:407] Connection header received
0619 14:41:29.796295 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:29.796298 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:29.796359 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:29.796456 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:29.797348 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:29.798019 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:29.798836 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:29.799087 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:29.799183 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:29.800492 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:29.800495 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:29.800498 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:41:29.800526 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:29.801876 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:29.802265 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:29.802269 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:29.802270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:29.802344 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:29.802579 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:29.802582 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:29.802584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:29.802757 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:29.802866 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:29.803091 (+ 225us) server_negotiation.cc:299] Negotiation successful
0619 14:41:29.803215 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 14:41:29.827939 21066 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:41:29.829339 21066 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:41:29.839426 21066 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b780cfa2defe47ac9cbd8e6cccae3e07: Generated new TSK 0
I20260619 14:41:29.840144 21066 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:41:29.852371 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 20984
2026-06-19T14:41:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1324 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781879837648036-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:41:29 dist-test-slave-32hj krb5kdc[21083](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:41:29 dist-test-slave-32hj krb5kdc[21083](info): set up 2 sockets
Jun 19 14:41:29 dist-test-slave-32hj krb5kdc[21083](info): commencing operation
krb5kdc: starting...
W20260619 14:41:30.808706 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781879837648036-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781879837648036-392-0/krb5kdc/test-user.keytab.
Jun 19 14:41:30 dist-test-slave-32hj krb5kdc[21083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:41:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:41:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781879837648036-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781879837648036-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:41:30.925706 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781879837648036-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781879837648036-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781879837648036-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42619
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781879837648036-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42619
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:41:31.040886 21096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:41:31.041189 21096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:41:31.041236 21096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:41:31.045301 21096 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:41:31.045390 21096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:41:31.045414 21096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:41:31.045432 21096 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:41:31.045449 21096 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:41:31.051024 21096 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781879837648036-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781879837648036-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42619
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781879837648036-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781879837648036-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:41:31.052142 21096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:41:31.053205 21096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:41:31.062098 21096 server_base.cc:1034] running on GCE node
W20260619 14:41:31.062139 21101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:31.062079 21104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:31.062139 21102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:31.063010 21096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:41:31.064255 21096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:41:31.065442 21096 hybrid_clock.cc:648] HybridClock initialized: now 1781880091065423 us; error 36 us; skew 500 ppm
Jun 19 14:41:31 dist-test-slave-32hj krb5kdc[21083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880091, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:41:31.068369 21096 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:41:31.069408 21096 webserver.cc:460] Webserver started at http://127.0.0.1:45577/ using document root <none> and password file <none>
I20260619 14:41:31.070003 21096 fs_manager.cc:359] Metadata directory not provided
I20260619 14:41:31.070079 21096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:41:31.070291 21096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:41:31.072515 21096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781879837648036-392-0/minicluster-data/master-0/data/instance:
uuid: "e5c52634ed08458f82a9a9c23548c0b5"
format_stamp: "Formatted at 2026-06-19 14:41:31 on dist-test-slave-32hj"
server_key: "07d0087d4135aa18496e05598f387626"
server_key_iv: "f3da1784f5b953bf3f80d920423e0d1c"
server_key_version: "encryptionkey@0"
I20260619 14:41:31.073155 21096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781879837648036-392-0/minicluster-data/master-0/wal/instance:
uuid: "e5c52634ed08458f82a9a9c23548c0b5"
format_stamp: "Formatted at 2026-06-19 14:41:31 on dist-test-slave-32hj"
server_key: "07d0087d4135aa18496e05598f387626"
server_key_iv: "f3da1784f5b953bf3f80d920423e0d1c"
server_key_version: "encryptionkey@0"
I20260619 14:41:31.077108 21096 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:41:31.079578 21111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:31.080864 21096 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:41:31.080986 21096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781879837648036-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781879837648036-392-0/minicluster-data/master-0/wal
uuid: "e5c52634ed08458f82a9a9c23548c0b5"
format_stamp: "Formatted at 2026-06-19 14:41:31 on dist-test-slave-32hj"
server_key: "07d0087d4135aa18496e05598f387626"
server_key_iv: "f3da1784f5b953bf3f80d920423e0d1c"
server_key_version: "encryptionkey@0"
I20260619 14:41:31.081091 21096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781879837648036-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781879837648036-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781879837648036-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:41:31.103621 21096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:41:31.106918 21096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:41:31.107115 21096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:41:31.114121 21096 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42619
I20260619 14:41:31.114144 21163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42619 every 8 connection(s)
I20260619 14:41:31.115427 21096 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:41:31.118227 21164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:41:31.122581 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 21096
I20260619 14:41:31.122714 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781879837648036-392-0/minicluster-data/master-0/wal/instance
I20260619 14:41:31.122956 392 external_mini_cluster.cc:1428] Setting key 2dfa22576b1f803263442f73a5125c0c
I20260619 14:41:31.124025 21164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5c52634ed08458f82a9a9c23548c0b5: Bootstrap starting.
I20260619 14:41:31.126319 21164 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5c52634ed08458f82a9a9c23548c0b5: Neither blocks nor log segments found. Creating new log.
I20260619 14:41:31.127064 21164 log.cc:824] T 00000000000000000000000000000000 P e5c52634ed08458f82a9a9c23548c0b5: Log is configured to *not* fsync() on all Append() calls
I20260619 14:41:31.129335 21164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5c52634ed08458f82a9a9c23548c0b5: No bootstrap required, opened a new log
I20260619 14:41:31.132057 21167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:31.124330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40583 (local address 127.0.0.1:42619)
0619 14:41:31.124776 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:31.124793 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:31.124829 (+ 36us) server_negotiation.cc:407] Connection header received
0619 14:41:31.125467 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:31.125485 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:31.125835 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:31.126131 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:31.126533 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:31.127657 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:31.128878 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:31.129400 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:31.129566 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:31.130379 (+ 813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:31.130385 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:31.130397 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:41:31.130805 (+ 408us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:31.130858 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:41:31.130881 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:31.130974 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:31.131114 (+ 140us) server_negotiation.cc:299] Negotiation successful
0619 14:41:31.131320 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":111,"threads_started":1}
I20260619 14:41:31.133029 21164 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5c52634ed08458f82a9a9c23548c0b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5c52634ed08458f82a9a9c23548c0b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42619 } }
I20260619 14:41:31.133248 21164 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5c52634ed08458f82a9a9c23548c0b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:41:31.133296 21164 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5c52634ed08458f82a9a9c23548c0b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c52634ed08458f82a9a9c23548c0b5, State: Initialized, Role: FOLLOWER
I20260619 14:41:31.133735 21164 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5c52634ed08458f82a9a9c23548c0b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5c52634ed08458f82a9a9c23548c0b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42619 } }
I20260619 14:41:31.133873 21164 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5c52634ed08458f82a9a9c23548c0b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:41:31.133939 21164 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5c52634ed08458f82a9a9c23548c0b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:41:31.134032 21164 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5c52634ed08458f82a9a9c23548c0b5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:41:31.134970 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:41:31.135169 21164 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5c52634ed08458f82a9a9c23548c0b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5c52634ed08458f82a9a9c23548c0b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42619 } }
I20260619 14:41:31.135536 21164 leader_election.cc:302] T 00000000000000000000000000000000 P e5c52634ed08458f82a9a9c23548c0b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5c52634ed08458f82a9a9c23548c0b5; no voters:
I20260619 14:41:31.135878 21164 leader_election.cc:288] T 00000000000000000000000000000000 P e5c52634ed08458f82a9a9c23548c0b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:41:31.135959 21169 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5c52634ed08458f82a9a9c23548c0b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:41:31.136164 21169 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5c52634ed08458f82a9a9c23548c0b5 [term 1 LEADER]: Becoming Leader. State: Replica: e5c52634ed08458f82a9a9c23548c0b5, State: Running, Role: LEADER
I20260619 14:41:31.136523 21169 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5c52634ed08458f82a9a9c23548c0b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5c52634ed08458f82a9a9c23548c0b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42619 } }
I20260619 14:41:31.137166 21164 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5c52634ed08458f82a9a9c23548c0b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:41:31.138114 21170 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5c52634ed08458f82a9a9c23548c0b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5c52634ed08458f82a9a9c23548c0b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5c52634ed08458f82a9a9c23548c0b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42619 } } }
I20260619 14:41:31.138232 21170 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5c52634ed08458f82a9a9c23548c0b5 [sys.catalog]: This master's current role is: LEADER
I20260619 14:41:31.138619 21183 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:41:31.138583 21169 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5c52634ed08458f82a9a9c23548c0b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5c52634ed08458f82a9a9c23548c0b5. Latest consensus state: current_term: 1 leader_uuid: "e5c52634ed08458f82a9a9c23548c0b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5c52634ed08458f82a9a9c23548c0b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42619 } } }
I20260619 14:41:31.138746 21169 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5c52634ed08458f82a9a9c23548c0b5 [sys.catalog]: This master's current role is: LEADER
I20260619 14:41:31.141777 21183 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:41:31.143765 21167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:31.138056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47595 (local address 127.0.0.1:42619)
0619 14:41:31.138159 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:31.138164 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:31.139420 (+ 1256us) server_negotiation.cc:407] Connection header received
0619 14:41:31.139607 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:31.139610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:31.139659 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:31.139734 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:31.140088 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:31.140854 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:31.141907 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:31.142211 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:31.142284 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:31.143328 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:31.143333 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:31.143336 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:41:31.143351 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:31.143383 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:41:31.143397 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:31.143476 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:31.143565 (+ 89us) server_negotiation.cc:299] Negotiation successful
0619 14:41:31.143625 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 14:41:31.147464 21183 catalog_manager.cc:1347] Generated new cluster ID: 4880ca521562484389b2a9acd302198f
I20260619 14:41:31.147537 21183 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:41:31.171232 21183 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:41:31.172233 21183 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:41:31.179545 21183 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5c52634ed08458f82a9a9c23548c0b5: Generated new TSK 0
I20260619 14:41:31.180322 21183 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:41:31.194031 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 21096
2026-06-19T14:41:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1341 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781879837648036-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:41:31 dist-test-slave-32hj krb5kdc[21195](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:41:31 dist-test-slave-32hj krb5kdc[21195](info): set up 2 sockets
Jun 19 14:41:31 dist-test-slave-32hj krb5kdc[21195](info): commencing operation
krb5kdc: starting...
W20260619 14:41:32.152848 392 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781879837648036-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781879837648036-392-0/krb5kdc/test-user.keytab.
Jun 19 14:41:32 dist-test-slave-32hj krb5kdc[21195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:41:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:41:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781879837648036-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781879837648036-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:41:32.288713 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781879837648036-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781879837648036-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781879837648036-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44615
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781879837648036-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44615
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:41:32.403612 21208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:41:32.403940 21208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:41:32.403987 21208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:41:32.408109 21208 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:41:32.408207 21208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:41:32.408229 21208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:41:32.408248 21208 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:41:32.408265 21208 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:41:32.413820 21208 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781879837648036-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781879837648036-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44615
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781879837648036-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781879837648036-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:41:32.414912 21208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:41:32.415915 21208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:41:32.424583 21214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:32.424683 21208 server_base.cc:1034] running on GCE node
W20260619 14:41:32.424520 21213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:32.424523 21216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:32.425436 21208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:41:32.426539 21208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:41:32.427732 21208 hybrid_clock.cc:648] HybridClock initialized: now 1781880092427715 us; error 32 us; skew 500 ppm
Jun 19 14:41:32 dist-test-slave-32hj krb5kdc[21195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:41:32.430510 21208 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:41:32.431417 21208 webserver.cc:460] Webserver started at http://127.0.0.1:43949/ using document root <none> and password file <none>
I20260619 14:41:32.431972 21208 fs_manager.cc:359] Metadata directory not provided
I20260619 14:41:32.432037 21208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:41:32.432236 21208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:41:32.434353 21208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781879837648036-392-0/minicluster-data/master-0/data/instance:
uuid: "510f8bc3d69b469f884e2970feb30a9f"
format_stamp: "Formatted at 2026-06-19 14:41:32 on dist-test-slave-32hj"
server_key: "ee3b6ad02ecdfe941aa740b8e3777552"
server_key_iv: "99d136f9a06d2467c48696c60c9b121e"
server_key_version: "encryptionkey@0"
I20260619 14:41:32.434932 21208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781879837648036-392-0/minicluster-data/master-0/wal/instance:
uuid: "510f8bc3d69b469f884e2970feb30a9f"
format_stamp: "Formatted at 2026-06-19 14:41:32 on dist-test-slave-32hj"
server_key: "ee3b6ad02ecdfe941aa740b8e3777552"
server_key_iv: "99d136f9a06d2467c48696c60c9b121e"
server_key_version: "encryptionkey@0"
I20260619 14:41:32.438761 21208 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:41:32.441290 21223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:32.442613 21208 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:41:32.442735 21208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781879837648036-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781879837648036-392-0/minicluster-data/master-0/wal
uuid: "510f8bc3d69b469f884e2970feb30a9f"
format_stamp: "Formatted at 2026-06-19 14:41:32 on dist-test-slave-32hj"
server_key: "ee3b6ad02ecdfe941aa740b8e3777552"
server_key_iv: "99d136f9a06d2467c48696c60c9b121e"
server_key_version: "encryptionkey@0"
I20260619 14:41:32.442842 21208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781879837648036-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781879837648036-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781879837648036-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:41:32.447614 21208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:41:32.450861 21208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:41:32.451081 21208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:41:32.457875 21208 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44615
I20260619 14:41:32.457922 21275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44615 every 8 connection(s)
I20260619 14:41:32.458848 21208 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:41:32.461910 21276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:41:32.465260 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 21208
I20260619 14:41:32.465427 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781879837648036-392-0/minicluster-data/master-0/wal/instance
I20260619 14:41:32.465709 392 external_mini_cluster.cc:1428] Setting key c41140fa04e7d4be308d6a92c95d5f78
I20260619 14:41:32.468909 21276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 510f8bc3d69b469f884e2970feb30a9f: Bootstrap starting.
I20260619 14:41:32.471540 21279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:32.467192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39509 (local address 127.0.0.1:44615)
0619 14:41:32.467688 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:32.467713 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:32.467759 (+ 46us) server_negotiation.cc:407] Connection header received
0619 14:41:32.468583 (+ 824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:32.468613 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:32.469032 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:32.469415 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:32.470122 (+ 707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:32.470128 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:32.470146 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:41:32.470157 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:32.470225 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:41:32.470252 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:32.470390 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:32.470470 (+ 80us) server_negotiation.cc:299] Negotiation successful
0619 14:41:32.470629 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":138,"threads_started":1}
I20260619 14:41:32.471709 21276 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 510f8bc3d69b469f884e2970feb30a9f: Neither blocks nor log segments found. Creating new log.
I20260619 14:41:32.472494 21276 log.cc:824] T 00000000000000000000000000000000 P 510f8bc3d69b469f884e2970feb30a9f: Log is configured to *not* fsync() on all Append() calls
I20260619 14:41:32.474476 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:41:32.474782 21276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 510f8bc3d69b469f884e2970feb30a9f: No bootstrap required, opened a new log
I20260619 14:41:32.478278 21279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:32.476442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36607 (local address 127.0.0.1:44615)
0619 14:41:32.476567 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:32.476570 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:32.476824 (+ 254us) server_negotiation.cc:407] Connection header received
0619 14:41:32.476959 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:32.476962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:32.476992 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:32.477056 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:32.477808 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:32.477811 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:32.477813 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:41:32.477823 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:32.477848 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:41:32.477858 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:32.477928 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:32.478105 (+ 177us) server_negotiation.cc:299] Negotiation successful
0619 14:41:32.478153 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 14:41:32.479069 21276 raft_consensus.cc:348] T 00000000000000000000000000000000 P 510f8bc3d69b469f884e2970feb30a9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "510f8bc3d69b469f884e2970feb30a9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44615 } }
I20260619 14:41:32.479369 21276 raft_consensus.cc:374] T 00000000000000000000000000000000 P 510f8bc3d69b469f884e2970feb30a9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:41:32.479441 21276 raft_consensus.cc:729] T 00000000000000000000000000000000 P 510f8bc3d69b469f884e2970feb30a9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 510f8bc3d69b469f884e2970feb30a9f, State: Initialized, Role: FOLLOWER
I20260619 14:41:32.479864 21276 consensus_queue.cc:260] T 00000000000000000000000000000000 P 510f8bc3d69b469f884e2970feb30a9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "510f8bc3d69b469f884e2970feb30a9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44615 } }
I20260619 14:41:32.479991 21276 raft_consensus.cc:388] T 00000000000000000000000000000000 P 510f8bc3d69b469f884e2970feb30a9f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:41:32.480055 21276 raft_consensus.cc:482] T 00000000000000000000000000000000 P 510f8bc3d69b469f884e2970feb30a9f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:41:32.480161 21276 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 510f8bc3d69b469f884e2970feb30a9f [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:41:32.481153 21276 raft_consensus.cc:504] T 00000000000000000000000000000000 P 510f8bc3d69b469f884e2970feb30a9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "510f8bc3d69b469f884e2970feb30a9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44615 } }
I20260619 14:41:32.481487 21276 leader_election.cc:302] T 00000000000000000000000000000000 P 510f8bc3d69b469f884e2970feb30a9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 510f8bc3d69b469f884e2970feb30a9f; no voters:
I20260619 14:41:32.481787 21276 leader_election.cc:288] T 00000000000000000000000000000000 P 510f8bc3d69b469f884e2970feb30a9f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:41:32.481884 21286 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 510f8bc3d69b469f884e2970feb30a9f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:41:32.482113 21286 raft_consensus.cc:686] T 00000000000000000000000000000000 P 510f8bc3d69b469f884e2970feb30a9f [term 1 LEADER]: Becoming Leader. State: Replica: 510f8bc3d69b469f884e2970feb30a9f, State: Running, Role: LEADER
I20260619 14:41:32.482460 21286 consensus_queue.cc:237] T 00000000000000000000000000000000 P 510f8bc3d69b469f884e2970feb30a9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "510f8bc3d69b469f884e2970feb30a9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44615 } }
I20260619 14:41:32.482956 21276 sys_catalog.cc:564] T 00000000000000000000000000000000 P 510f8bc3d69b469f884e2970feb30a9f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:41:32.484087 21288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 510f8bc3d69b469f884e2970feb30a9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 510f8bc3d69b469f884e2970feb30a9f. Latest consensus state: current_term: 1 leader_uuid: "510f8bc3d69b469f884e2970feb30a9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "510f8bc3d69b469f884e2970feb30a9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44615 } } }
I20260619 14:41:32.484220 21288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 510f8bc3d69b469f884e2970feb30a9f [sys.catalog]: This master's current role is: LEADER
I20260619 14:41:32.484081 21287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 510f8bc3d69b469f884e2970feb30a9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "510f8bc3d69b469f884e2970feb30a9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "510f8bc3d69b469f884e2970feb30a9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44615 } } }
I20260619 14:41:32.484431 21287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 510f8bc3d69b469f884e2970feb30a9f [sys.catalog]: This master's current role is: LEADER
I20260619 14:41:32.484634 21295 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:41:32.487977 21295 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:41:32.493409 21295 catalog_manager.cc:1347] Generated new cluster ID: afe1651fbf0c4ad59ad739b437b01236
I20260619 14:41:32.493497 21295 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:41:32.503533 21295 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:41:32.503898 21295 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:41:32.516659 21295 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 510f8bc3d69b469f884e2970feb30a9f: Generated new TSK 0
I20260619 14:41:32.517663 21295 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:41:32.526851 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 21208
2026-06-19T14:41:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1332 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781879837648036-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:41:32 dist-test-slave-32hj krb5kdc[21307](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:41:32 dist-test-slave-32hj krb5kdc[21307](info): set up 2 sockets
Jun 19 14:41:32 dist-test-slave-32hj krb5kdc[21307](info): commencing operation
krb5kdc: starting...
W20260619 14:41:33.484786 392 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781879837648036-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781879837648036-392-0/krb5kdc/test-user.keytab.
Jun 19 14:41:33 dist-test-slave-32hj krb5kdc[21307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:41:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:41:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781879837648036-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781879837648036-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:41:33.608283 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781879837648036-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781879837648036-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781879837648036-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42361
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781879837648036-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42361
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:41:33.718920 21320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:41:33.719200 21320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:41:33.719241 21320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:41:33.723296 21320 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:41:33.723384 21320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:41:33.723405 21320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:41:33.723423 21320 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:41:33.723441 21320 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:41:33.728871 21320 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781879837648036-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781879837648036-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42361
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781879837648036-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781879837648036-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:41:33.729943 21320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:41:33.730911 21320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:41:33.739563 21320 server_base.cc:1034] running on GCE node
W20260619 14:41:33.739514 21328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:33.739773 21325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:33.739511 21326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:33.740445 21320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:41:33.741639 21320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:41:33.742808 21320 hybrid_clock.cc:648] HybridClock initialized: now 1781880093742792 us; error 29 us; skew 500 ppm
Jun 19 14:41:33 dist-test-slave-32hj krb5kdc[21307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880093, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:41:33.749272 21320 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:41:33.750439 21320 webserver.cc:460] Webserver started at http://127.0.0.1:33547/ using document root <none> and password file <none>
I20260619 14:41:33.751016 21320 fs_manager.cc:359] Metadata directory not provided
I20260619 14:41:33.751089 21320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:41:33.751298 21320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:41:33.753528 21320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781879837648036-392-0/minicluster-data/master-0/data/instance:
uuid: "278c17641c6743c3b039192cded609f5"
format_stamp: "Formatted at 2026-06-19 14:41:33 on dist-test-slave-32hj"
server_key: "1f5bf3b7899301c2a37713564782860a"
server_key_iv: "11a7e527f222275e0d247d4158b0d8bf"
server_key_version: "encryptionkey@0"
I20260619 14:41:33.754079 21320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781879837648036-392-0/minicluster-data/master-0/wal/instance:
uuid: "278c17641c6743c3b039192cded609f5"
format_stamp: "Formatted at 2026-06-19 14:41:33 on dist-test-slave-32hj"
server_key: "1f5bf3b7899301c2a37713564782860a"
server_key_iv: "11a7e527f222275e0d247d4158b0d8bf"
server_key_version: "encryptionkey@0"
I20260619 14:41:33.757884 21320 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260619 14:41:33.760432 21335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:33.761943 21320 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:41:33.762084 21320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781879837648036-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781879837648036-392-0/minicluster-data/master-0/wal
uuid: "278c17641c6743c3b039192cded609f5"
format_stamp: "Formatted at 2026-06-19 14:41:33 on dist-test-slave-32hj"
server_key: "1f5bf3b7899301c2a37713564782860a"
server_key_iv: "11a7e527f222275e0d247d4158b0d8bf"
server_key_version: "encryptionkey@0"
I20260619 14:41:33.762197 21320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781879837648036-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781879837648036-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781879837648036-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:41:33.774021 21320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:41:33.777258 21320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:41:33.777442 21320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:41:33.784276 21320 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42361
I20260619 14:41:33.784303 21387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42361 every 8 connection(s)
I20260619 14:41:33.785317 21320 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:41:33.788190 21388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:41:33.794020 21388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 278c17641c6743c3b039192cded609f5: Bootstrap starting.
I20260619 14:41:33.794564 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 21320
I20260619 14:41:33.794700 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781879837648036-392-0/minicluster-data/master-0/wal/instance
I20260619 14:41:33.794960 392 external_mini_cluster.cc:1428] Setting key 3571d99da3b92be8895d397c6da8ac20
I20260619 14:41:33.797003 21388 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 278c17641c6743c3b039192cded609f5: Neither blocks nor log segments found. Creating new log.
I20260619 14:41:33.797785 21388 log.cc:824] T 00000000000000000000000000000000 P 278c17641c6743c3b039192cded609f5: Log is configured to *not* fsync() on all Append() calls
I20260619 14:41:33.799755 21388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 278c17641c6743c3b039192cded609f5: No bootstrap required, opened a new log
I20260619 14:41:33.803608 21388 raft_consensus.cc:348] T 00000000000000000000000000000000 P 278c17641c6743c3b039192cded609f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "278c17641c6743c3b039192cded609f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42361 } }
I20260619 14:41:33.803933 21388 raft_consensus.cc:374] T 00000000000000000000000000000000 P 278c17641c6743c3b039192cded609f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:41:33.803997 21388 raft_consensus.cc:729] T 00000000000000000000000000000000 P 278c17641c6743c3b039192cded609f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 278c17641c6743c3b039192cded609f5, State: Initialized, Role: FOLLOWER
Jun 19 14:41:33 dist-test-slave-32hj krb5kdc[21307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 14:41:33.804550 21388 consensus_queue.cc:260] T 00000000000000000000000000000000 P 278c17641c6743c3b039192cded609f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "278c17641c6743c3b039192cded609f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42361 } }
I20260619 14:41:33.804749 21388 raft_consensus.cc:388] T 00000000000000000000000000000000 P 278c17641c6743c3b039192cded609f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:41:33.804816 21388 raft_consensus.cc:482] T 00000000000000000000000000000000 P 278c17641c6743c3b039192cded609f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:41:33.804921 21388 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 278c17641c6743c3b039192cded609f5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:41:33.806005 21388 raft_consensus.cc:504] T 00000000000000000000000000000000 P 278c17641c6743c3b039192cded609f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "278c17641c6743c3b039192cded609f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42361 } }
I20260619 14:41:33.806356 21388 leader_election.cc:302] T 00000000000000000000000000000000 P 278c17641c6743c3b039192cded609f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 278c17641c6743c3b039192cded609f5; no voters:
I20260619 14:41:33.806680 21388 leader_election.cc:288] T 00000000000000000000000000000000 P 278c17641c6743c3b039192cded609f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:41:33.806813 21393 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 278c17641c6743c3b039192cded609f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:41:33.807030 21393 raft_consensus.cc:686] T 00000000000000000000000000000000 P 278c17641c6743c3b039192cded609f5 [term 1 LEADER]: Becoming Leader. State: Replica: 278c17641c6743c3b039192cded609f5, State: Running, Role: LEADER
I20260619 14:41:33.807387 21393 consensus_queue.cc:237] T 00000000000000000000000000000000 P 278c17641c6743c3b039192cded609f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "278c17641c6743c3b039192cded609f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42361 } }
I20260619 14:41:33.808030 21388 sys_catalog.cc:564] T 00000000000000000000000000000000 P 278c17641c6743c3b039192cded609f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:41:33.808605 21393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 278c17641c6743c3b039192cded609f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 278c17641c6743c3b039192cded609f5. Latest consensus state: current_term: 1 leader_uuid: "278c17641c6743c3b039192cded609f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "278c17641c6743c3b039192cded609f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42361 } } }
I20260619 14:41:33.808808 21393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 278c17641c6743c3b039192cded609f5 [sys.catalog]: This master's current role is: LEADER
I20260619 14:41:33.809036 21394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 278c17641c6743c3b039192cded609f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "278c17641c6743c3b039192cded609f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "278c17641c6743c3b039192cded609f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42361 } } }
I20260619 14:41:33.809265 21397 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:41:33.810761 21394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 278c17641c6743c3b039192cded609f5 [sys.catalog]: This master's current role is: LEADER
I20260619 14:41:33.811275 21391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:33.796427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.157:52745 (local address 127.0.0.1:42361)
0619 14:41:33.796992 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:33.797009 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:33.797057 (+ 48us) server_negotiation.cc:407] Connection header received
0619 14:41:33.797777 (+ 720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:33.797795 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:33.798211 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:33.798547 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:33.799358 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:33.800380 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:33.801338 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:33.801848 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:33.802002 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:33.805023 (+ 3021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:33.805031 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:33.805050 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:41:33.805086 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:33.806995 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:33.807668 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:33.807674 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:33.807681 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:33.807831 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:33.808233 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:33.808237 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:33.808239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:33.808862 (+ 623us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:33.809823 (+ 961us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:33.810009 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 14:41:33.810244 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":401,"thread_start_us":146,"threads_started":1}
I20260619 14:41:33.813977 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:41:33.814103 21397 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:41:33.819792 21397 catalog_manager.cc:1347] Generated new cluster ID: de3eed0ec7534eef8d2f0a1cba49ca95
I20260619 14:41:33.819854 21397 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:41:33.824390 21391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:33.815817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.157:44831 (local address 127.0.0.1:42361)
0619 14:41:33.815919 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:33.815923 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:33.816280 (+ 357us) server_negotiation.cc:407] Connection header received
0619 14:41:33.816410 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:33.816413 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:33.816467 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:33.816539 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:33.817354 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:33.817995 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:33.818762 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:33.819007 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:33.819091 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:33.820658 (+ 1567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:33.820662 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:33.820665 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:41:33.820689 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:33.822276 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:33.822761 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:33.822764 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:33.822766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:33.822835 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:33.823174 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:33.823177 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:33.823179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:33.823353 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:33.823477 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:33.824150 (+ 673us) server_negotiation.cc:299] Negotiation successful
0619 14:41:33.824236 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 14:41:33.828639 21397 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:41:33.829916 21397 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:41:33.846276 21397 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 278c17641c6743c3b039192cded609f5: Generated new TSK 0
I20260619 14:41:33.847087 21397 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:41:33.851639 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 21320
W20260619 14:41:33.856168 21305 connection.cc:535] client connection to 127.0.0.1:42361 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42361): Connection reset by peer (error 104)
2026-06-19T14:41:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1324 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781879837648036-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:41:33 dist-test-slave-32hj krb5kdc[21419](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:41:33 dist-test-slave-32hj krb5kdc[21419](info): set up 2 sockets
Jun 19 14:41:33 dist-test-slave-32hj krb5kdc[21419](info): commencing operation
krb5kdc: starting...
W20260619 14:41:34.812947 392 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781879837648036-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781879837648036-392-0/krb5kdc/test-user.keytab.
Jun 19 14:41:34 dist-test-slave-32hj krb5kdc[21419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:41:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:41:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781879837648036-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781879837648036-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:41:34.931929 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781879837648036-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781879837648036-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781879837648036-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38137
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781879837648036-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38137
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:41:35.043196 21432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:41:35.043509 21432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:41:35.043572 21432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:41:35.047699 21432 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:41:35.047775 21432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:41:35.047802 21432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:41:35.047823 21432 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:41:35.047845 21432 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:41:35.053975 21432 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781879837648036-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781879837648036-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38137
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781879837648036-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781879837648036-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:41:35.055130 21432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:41:35.056177 21432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:41:35.064862 21440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:35.064875 21438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:35.064915 21432 server_base.cc:1034] running on GCE node
W20260619 14:41:35.064863 21437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:35.065701 21432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:41:35.066841 21432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:41:35.068003 21432 hybrid_clock.cc:648] HybridClock initialized: now 1781880095067982 us; error 32 us; skew 500 ppm
Jun 19 14:41:35 dist-test-slave-32hj krb5kdc[21419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:41:35.070704 21432 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:41:35.071635 21432 webserver.cc:460] Webserver started at http://127.0.0.1:37747/ using document root <none> and password file <none>
I20260619 14:41:35.072196 21432 fs_manager.cc:359] Metadata directory not provided
I20260619 14:41:35.072263 21432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:41:35.072479 21432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:41:35.074625 21432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781879837648036-392-0/minicluster-data/master-0/data/instance:
uuid: "e48f06b155d74f8198d48ed959e78802"
format_stamp: "Formatted at 2026-06-19 14:41:35 on dist-test-slave-32hj"
server_key: "df7108cfbe43896a6d2706a7c9cb94b4"
server_key_iv: "674d1fa7411e2d657b0393336bdb708e"
server_key_version: "encryptionkey@0"
I20260619 14:41:35.075191 21432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781879837648036-392-0/minicluster-data/master-0/wal/instance:
uuid: "e48f06b155d74f8198d48ed959e78802"
format_stamp: "Formatted at 2026-06-19 14:41:35 on dist-test-slave-32hj"
server_key: "df7108cfbe43896a6d2706a7c9cb94b4"
server_key_iv: "674d1fa7411e2d657b0393336bdb708e"
server_key_version: "encryptionkey@0"
I20260619 14:41:35.079022 21432 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:41:35.081633 21447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:35.082921 21432 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:41:35.083071 21432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781879837648036-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781879837648036-392-0/minicluster-data/master-0/wal
uuid: "e48f06b155d74f8198d48ed959e78802"
format_stamp: "Formatted at 2026-06-19 14:41:35 on dist-test-slave-32hj"
server_key: "df7108cfbe43896a6d2706a7c9cb94b4"
server_key_iv: "674d1fa7411e2d657b0393336bdb708e"
server_key_version: "encryptionkey@0"
I20260619 14:41:35.083178 21432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781879837648036-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781879837648036-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781879837648036-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:41:35.102268 21432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:41:35.105623 21432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:41:35.105813 21432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:41:35.112686 21432 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38137
I20260619 14:41:35.112740 21499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38137 every 8 connection(s)
I20260619 14:41:35.113708 21432 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:41:35.116581 21500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:41:35.118037 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 21432
I20260619 14:41:35.118180 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781879837648036-392-0/minicluster-data/master-0/wal/instance
I20260619 14:41:35.118462 392 external_mini_cluster.cc:1428] Setting key f55b22e59469a340470d2c8de3e1be9e
I20260619 14:41:35.123415 21500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e48f06b155d74f8198d48ed959e78802: Bootstrap starting.
I20260619 14:41:35.126214 21500 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e48f06b155d74f8198d48ed959e78802: Neither blocks nor log segments found. Creating new log.
I20260619 14:41:35.127004 21500 log.cc:824] T 00000000000000000000000000000000 P e48f06b155d74f8198d48ed959e78802: Log is configured to *not* fsync() on all Append() calls
I20260619 14:41:35.128351 21503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:35.119953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.157:50763 (local address 127.0.0.1:38137)
0619 14:41:35.120387 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:35.120404 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:35.120440 (+ 36us) server_negotiation.cc:407] Connection header received
0619 14:41:35.121129 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:35.121148 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:35.121510 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:35.121812 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:35.122379 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:35.123478 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:35.124694 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:35.125260 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:35.125397 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:35.126424 (+ 1027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:35.126430 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:35.126445 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:41:35.126909 (+ 464us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:35.126978 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:41:35.127006 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:35.127147 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:35.127362 (+ 215us) server_negotiation.cc:299] Negotiation successful
0619 14:41:35.127513 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":117,"threads_started":1}
I20260619 14:41:35.129071 21500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e48f06b155d74f8198d48ed959e78802: No bootstrap required, opened a new log
I20260619 14:41:35.131464 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:41:35.133407 21500 raft_consensus.cc:348] T 00000000000000000000000000000000 P e48f06b155d74f8198d48ed959e78802 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e48f06b155d74f8198d48ed959e78802" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38137 } }
I20260619 14:41:35.133688 21500 raft_consensus.cc:374] T 00000000000000000000000000000000 P e48f06b155d74f8198d48ed959e78802 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:41:35.133797 21500 raft_consensus.cc:729] T 00000000000000000000000000000000 P e48f06b155d74f8198d48ed959e78802 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e48f06b155d74f8198d48ed959e78802, State: Initialized, Role: FOLLOWER
I20260619 14:41:35.134326 21500 consensus_queue.cc:260] T 00000000000000000000000000000000 P e48f06b155d74f8198d48ed959e78802 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e48f06b155d74f8198d48ed959e78802" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38137 } }
I20260619 14:41:35.134493 21500 raft_consensus.cc:388] T 00000000000000000000000000000000 P e48f06b155d74f8198d48ed959e78802 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:41:35.134569 21500 raft_consensus.cc:482] T 00000000000000000000000000000000 P e48f06b155d74f8198d48ed959e78802 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:41:35.134675 21500 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e48f06b155d74f8198d48ed959e78802 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:41:35.135794 21500 raft_consensus.cc:504] T 00000000000000000000000000000000 P e48f06b155d74f8198d48ed959e78802 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e48f06b155d74f8198d48ed959e78802" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38137 } }
I20260619 14:41:35.136179 21500 leader_election.cc:302] T 00000000000000000000000000000000 P e48f06b155d74f8198d48ed959e78802 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e48f06b155d74f8198d48ed959e78802; no voters:
I20260619 14:41:35.136518 21500 leader_election.cc:288] T 00000000000000000000000000000000 P e48f06b155d74f8198d48ed959e78802 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:41:35.136601 21510 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e48f06b155d74f8198d48ed959e78802 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:41:35.136863 21510 raft_consensus.cc:686] T 00000000000000000000000000000000 P e48f06b155d74f8198d48ed959e78802 [term 1 LEADER]: Becoming Leader. State: Replica: e48f06b155d74f8198d48ed959e78802, State: Running, Role: LEADER
I20260619 14:41:35.137217 21510 consensus_queue.cc:237] T 00000000000000000000000000000000 P e48f06b155d74f8198d48ed959e78802 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e48f06b155d74f8198d48ed959e78802" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38137 } }
I20260619 14:41:35.137943 21503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:35.133734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.157:36467 (local address 127.0.0.1:38137)
0619 14:41:35.133871 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:35.133875 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:35.134044 (+ 169us) server_negotiation.cc:407] Connection header received
0619 14:41:35.134178 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:35.134181 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:35.134224 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:35.134281 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:35.134562 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:35.135152 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:35.136192 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:35.136467 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:35.136555 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:35.137498 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:35.137502 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:35.137504 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:41:35.137517 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:35.137543 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:41:35.137553 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:35.137624 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:35.137764 (+ 140us) server_negotiation.cc:299] Negotiation successful
0619 14:41:35.137798 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 14:41:35.137972 21500 sys_catalog.cc:564] T 00000000000000000000000000000000 P e48f06b155d74f8198d48ed959e78802 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:41:35.139719 21511 sys_catalog.cc:455] T 00000000000000000000000000000000 P e48f06b155d74f8198d48ed959e78802 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e48f06b155d74f8198d48ed959e78802. Latest consensus state: current_term: 1 leader_uuid: "e48f06b155d74f8198d48ed959e78802" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e48f06b155d74f8198d48ed959e78802" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38137 } } }
I20260619 14:41:35.139853 21511 sys_catalog.cc:458] T 00000000000000000000000000000000 P e48f06b155d74f8198d48ed959e78802 [sys.catalog]: This master's current role is: LEADER
I20260619 14:41:35.140187 21525 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:41:35.140161 21512 sys_catalog.cc:455] T 00000000000000000000000000000000 P e48f06b155d74f8198d48ed959e78802 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e48f06b155d74f8198d48ed959e78802" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e48f06b155d74f8198d48ed959e78802" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38137 } } }
I20260619 14:41:35.140607 21512 sys_catalog.cc:458] T 00000000000000000000000000000000 P e48f06b155d74f8198d48ed959e78802 [sys.catalog]: This master's current role is: LEADER
I20260619 14:41:35.142854 21525 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:41:35.147940 21525 catalog_manager.cc:1347] Generated new cluster ID: 7623730ee7a449dd97d1dad9d72e66ad
I20260619 14:41:35.148010 21525 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:41:35.162359 21525 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:41:35.163376 21525 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:41:35.170506 21525 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e48f06b155d74f8198d48ed959e78802: Generated new TSK 0
I20260619 14:41:35.171475 21525 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:41:35.175768 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 21432
W20260619 14:41:35.180166 21417 connection.cc:535] client connection to 127.0.0.1:38137 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38137): Connection reset by peer (error 104)
2026-06-19T14:41:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1323 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781879837648036-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:41:35 dist-test-slave-32hj krb5kdc[21531](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:41:35 dist-test-slave-32hj krb5kdc[21531](info): set up 2 sockets
Jun 19 14:41:35 dist-test-slave-32hj krb5kdc[21531](info): commencing operation
krb5kdc: starting...
W20260619 14:41:36.124639 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781879837648036-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781879837648036-392-0/krb5kdc/test-user.keytab.
Jun 19 14:41:36 dist-test-slave-32hj krb5kdc[21531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:41:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:41:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781879837648036-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781879837648036-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:41:36.243911 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781879837648036-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781879837648036-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781879837648036-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46545
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781879837648036-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46545
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:41:36.353293 21544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:41:36.353610 21544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:41:36.353672 21544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:41:36.357779 21544 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:41:36.357860 21544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:41:36.357888 21544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:41:36.357910 21544 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:41:36.357937 21544 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:41:36.363466 21544 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781879837648036-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781879837648036-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46545
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781879837648036-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781879837648036-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:41:36.364602 21544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:41:36.365681 21544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:41:36.374115 21552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:36.374106 21549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:36.374106 21544 server_base.cc:1034] running on GCE node
W20260619 14:41:36.374106 21550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:36.374946 21544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:41:36.376076 21544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:41:36.377238 21544 hybrid_clock.cc:648] HybridClock initialized: now 1781880096377219 us; error 35 us; skew 500 ppm
Jun 19 14:41:36 dist-test-slave-32hj krb5kdc[21531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880096, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:41:36.380087 21544 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:41:36.381094 21544 webserver.cc:460] Webserver started at http://127.0.0.1:38701/ using document root <none> and password file <none>
I20260619 14:41:36.381645 21544 fs_manager.cc:359] Metadata directory not provided
I20260619 14:41:36.381707 21544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:41:36.381896 21544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:41:36.383936 21544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781879837648036-392-0/minicluster-data/master-0/data/instance:
uuid: "9a731905eb7c4539a4e61023c52d84d0"
format_stamp: "Formatted at 2026-06-19 14:41:36 on dist-test-slave-32hj"
server_key: "de3f40a9f8e840d30d1bbb3864251d61"
server_key_iv: "cec9f9b7701733cf7f2bb44dfda651a2"
server_key_version: "encryptionkey@0"
I20260619 14:41:36.384478 21544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781879837648036-392-0/minicluster-data/master-0/wal/instance:
uuid: "9a731905eb7c4539a4e61023c52d84d0"
format_stamp: "Formatted at 2026-06-19 14:41:36 on dist-test-slave-32hj"
server_key: "de3f40a9f8e840d30d1bbb3864251d61"
server_key_iv: "cec9f9b7701733cf7f2bb44dfda651a2"
server_key_version: "encryptionkey@0"
I20260619 14:41:36.388253 21544 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 14:41:36.390709 21559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:36.391942 21544 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:41:36.392105 21544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781879837648036-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781879837648036-392-0/minicluster-data/master-0/wal
uuid: "9a731905eb7c4539a4e61023c52d84d0"
format_stamp: "Formatted at 2026-06-19 14:41:36 on dist-test-slave-32hj"
server_key: "de3f40a9f8e840d30d1bbb3864251d61"
server_key_iv: "cec9f9b7701733cf7f2bb44dfda651a2"
server_key_version: "encryptionkey@0"
I20260619 14:41:36.392210 21544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781879837648036-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781879837648036-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781879837648036-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:41:36.396710 21544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:41:36.399854 21544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:41:36.400030 21544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:41:36.406759 21544 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46545
I20260619 14:41:36.406776 21611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46545 every 8 connection(s)
I20260619 14:41:36.407701 21544 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:41:36.409801 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 21544
I20260619 14:41:36.409931 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781879837648036-392-0/minicluster-data/master-0/wal/instance
I20260619 14:41:36.410218 392 external_mini_cluster.cc:1428] Setting key f4156a83d2c26af9273191124e0f374b
I20260619 14:41:36.410712 21612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:41:36.415398 21615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:36.411565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.157:45069 (local address 127.0.0.1:46545)
0619 14:41:36.411949 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:36.411965 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:36.412001 (+ 36us) server_negotiation.cc:407] Connection header received
0619 14:41:36.412665 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:36.412681 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:36.413015 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:36.413307 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:36.414179 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:36.414184 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:36.414196 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:41:36.414206 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:36.414260 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:41:36.414280 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:36.414391 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:36.414505 (+ 114us) server_negotiation.cc:299] Negotiation successful
0619 14:41:36.414655 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":102,"threads_started":1}
I20260619 14:41:36.417404 21612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a731905eb7c4539a4e61023c52d84d0: Bootstrap starting.
I20260619 14:41:36.418218 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:41:36.420023 21612 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a731905eb7c4539a4e61023c52d84d0: Neither blocks nor log segments found. Creating new log.
I20260619 14:41:36.420872 21612 log.cc:824] T 00000000000000000000000000000000 P 9a731905eb7c4539a4e61023c52d84d0: Log is configured to *not* fsync() on all Append() calls
I20260619 14:41:36.422224 21615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:36.420411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.157:42067 (local address 127.0.0.1:46545)
0619 14:41:36.420522 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:36.420526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:36.420973 (+ 447us) server_negotiation.cc:407] Connection header received
0619 14:41:36.421105 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:36.421108 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:36.421142 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:36.421229 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:36.421887 (+ 658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:36.421891 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:36.421893 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:41:36.421905 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:36.421933 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:41:36.421945 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:36.422028 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:36.422076 (+ 48us) server_negotiation.cc:299] Negotiation successful
0619 14:41:36.422113 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 14:41:36.423082 21612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a731905eb7c4539a4e61023c52d84d0: No bootstrap required, opened a new log
I20260619 14:41:36.426458 21612 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a731905eb7c4539a4e61023c52d84d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a731905eb7c4539a4e61023c52d84d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46545 } }
I20260619 14:41:36.426678 21612 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a731905eb7c4539a4e61023c52d84d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:41:36.426712 21612 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a731905eb7c4539a4e61023c52d84d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a731905eb7c4539a4e61023c52d84d0, State: Initialized, Role: FOLLOWER
I20260619 14:41:36.427127 21612 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a731905eb7c4539a4e61023c52d84d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a731905eb7c4539a4e61023c52d84d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46545 } }
I20260619 14:41:36.427245 21612 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a731905eb7c4539a4e61023c52d84d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:41:36.427316 21612 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a731905eb7c4539a4e61023c52d84d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:41:36.427415 21612 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a731905eb7c4539a4e61023c52d84d0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:41:36.428293 21612 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a731905eb7c4539a4e61023c52d84d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a731905eb7c4539a4e61023c52d84d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46545 } }
I20260619 14:41:36.428601 21612 leader_election.cc:302] T 00000000000000000000000000000000 P 9a731905eb7c4539a4e61023c52d84d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a731905eb7c4539a4e61023c52d84d0; no voters:
I20260619 14:41:36.428979 21612 leader_election.cc:288] T 00000000000000000000000000000000 P 9a731905eb7c4539a4e61023c52d84d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:41:36.429075 21622 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a731905eb7c4539a4e61023c52d84d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:41:36.429296 21622 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a731905eb7c4539a4e61023c52d84d0 [term 1 LEADER]: Becoming Leader. State: Replica: 9a731905eb7c4539a4e61023c52d84d0, State: Running, Role: LEADER
I20260619 14:41:36.429628 21622 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a731905eb7c4539a4e61023c52d84d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a731905eb7c4539a4e61023c52d84d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46545 } }
I20260619 14:41:36.430169 21612 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a731905eb7c4539a4e61023c52d84d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:41:36.431174 21623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a731905eb7c4539a4e61023c52d84d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a731905eb7c4539a4e61023c52d84d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a731905eb7c4539a4e61023c52d84d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46545 } } }
I20260619 14:41:36.431331 21623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a731905eb7c4539a4e61023c52d84d0 [sys.catalog]: This master's current role is: LEADER
I20260619 14:41:36.431584 21624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a731905eb7c4539a4e61023c52d84d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a731905eb7c4539a4e61023c52d84d0. Latest consensus state: current_term: 1 leader_uuid: "9a731905eb7c4539a4e61023c52d84d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a731905eb7c4539a4e61023c52d84d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46545 } } }
I20260619 14:41:36.431665 21624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a731905eb7c4539a4e61023c52d84d0 [sys.catalog]: This master's current role is: LEADER
I20260619 14:41:36.431686 21631 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:41:36.434749 21631 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:41:36.440150 21631 catalog_manager.cc:1347] Generated new cluster ID: 65a8a511e7f84648bf03e15d99f8721b
I20260619 14:41:36.440223 21631 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:41:36.471001 21631 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:41:36.471364 21631 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:41:36.491616 21631 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a731905eb7c4539a4e61023c52d84d0: Generated new TSK 0
I20260619 14:41:36.492588 21631 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:41:36.505662 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 21544
2026-06-19T14:41:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1330 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781879837648036-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:41:36 dist-test-slave-32hj krb5kdc[21643](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:41:36 dist-test-slave-32hj krb5kdc[21643](info): set up 2 sockets
Jun 19 14:41:36 dist-test-slave-32hj krb5kdc[21643](info): commencing operation
krb5kdc: starting...
W20260619 14:41:37.465095 392 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781879837648036-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781879837648036-392-0/krb5kdc/test-user.keytab.
Jun 19 14:41:37 dist-test-slave-32hj krb5kdc[21643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880097, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:41:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:41:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781879837648036-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781879837648036-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:41:37.582973 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781879837648036-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781879837648036-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781879837648036-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45089
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781879837648036-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45089
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:41:37.689947 21656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:41:37.690234 21656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:41:37.690294 21656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:41:37.694283 21656 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:41:37.694348 21656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:41:37.694375 21656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:41:37.694394 21656 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:41:37.694415 21656 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:41:37.699751 21656 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781879837648036-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781879837648036-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45089
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781879837648036-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781879837648036-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:41:37.700850 21656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:41:37.701861 21656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:41:37.710009 21664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:37.710018 21662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:37.710026 21661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:37.710346 21656 server_base.cc:1034] running on GCE node
I20260619 14:41:37.710898 21656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:41:37.711987 21656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:41:37.713146 21656 hybrid_clock.cc:648] HybridClock initialized: now 1781880097713114 us; error 44 us; skew 500 ppm
Jun 19 14:41:37 dist-test-slave-32hj krb5kdc[21643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880097, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:41:37.715888 21656 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:41:37.716861 21656 webserver.cc:460] Webserver started at http://127.0.0.1:41355/ using document root <none> and password file <none>
I20260619 14:41:37.717438 21656 fs_manager.cc:359] Metadata directory not provided
I20260619 14:41:37.717507 21656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:41:37.717715 21656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:41:37.719796 21656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781879837648036-392-0/minicluster-data/master-0/data/instance:
uuid: "ea1fec84dee44082917ebf4bff172291"
format_stamp: "Formatted at 2026-06-19 14:41:37 on dist-test-slave-32hj"
server_key: "83b5842836a49ac0a43944f15ac7c05e"
server_key_iv: "c5f084bb0c59fa277b1576d5422a085e"
server_key_version: "encryptionkey@0"
I20260619 14:41:37.720371 21656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781879837648036-392-0/minicluster-data/master-0/wal/instance:
uuid: "ea1fec84dee44082917ebf4bff172291"
format_stamp: "Formatted at 2026-06-19 14:41:37 on dist-test-slave-32hj"
server_key: "83b5842836a49ac0a43944f15ac7c05e"
server_key_iv: "c5f084bb0c59fa277b1576d5422a085e"
server_key_version: "encryptionkey@0"
I20260619 14:41:37.724193 21656 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260619 14:41:37.726868 21671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:37.728191 21656 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:41:37.728336 21656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781879837648036-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781879837648036-392-0/minicluster-data/master-0/wal
uuid: "ea1fec84dee44082917ebf4bff172291"
format_stamp: "Formatted at 2026-06-19 14:41:37 on dist-test-slave-32hj"
server_key: "83b5842836a49ac0a43944f15ac7c05e"
server_key_iv: "c5f084bb0c59fa277b1576d5422a085e"
server_key_version: "encryptionkey@0"
I20260619 14:41:37.728456 21656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781879837648036-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781879837648036-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781879837648036-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:41:37.742419 21656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:41:37.745672 21656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:41:37.745885 21656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:41:37.752571 21656 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45089
I20260619 14:41:37.752625 21723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45089 every 8 connection(s)
I20260619 14:41:37.753575 21656 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:41:37.756508 21724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:41:37.758914 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 21656
I20260619 14:41:37.759052 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781879837648036-392-0/minicluster-data/master-0/wal/instance
I20260619 14:41:37.759331 392 external_mini_cluster.cc:1428] Setting key a99fae021c8eb0ea8e136edb70edea74
I20260619 14:41:37.763090 21724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea1fec84dee44082917ebf4bff172291: Bootstrap starting.
I20260619 14:41:37.765786 21724 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea1fec84dee44082917ebf4bff172291: Neither blocks nor log segments found. Creating new log.
I20260619 14:41:37.766640 21724 log.cc:824] T 00000000000000000000000000000000 P ea1fec84dee44082917ebf4bff172291: Log is configured to *not* fsync() on all Append() calls
I20260619 14:41:37.768674 21724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea1fec84dee44082917ebf4bff172291: No bootstrap required, opened a new log
Jun 19 14:41:37 dist-test-slave-32hj krb5kdc[21643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880097, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 14:41:37.772603 21724 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea1fec84dee44082917ebf4bff172291 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea1fec84dee44082917ebf4bff172291" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45089 } }
I20260619 14:41:37.772902 21724 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea1fec84dee44082917ebf4bff172291 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:41:37.772997 21724 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea1fec84dee44082917ebf4bff172291 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea1fec84dee44082917ebf4bff172291, State: Initialized, Role: FOLLOWER
I20260619 14:41:37.773492 21724 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea1fec84dee44082917ebf4bff172291 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea1fec84dee44082917ebf4bff172291" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45089 } }
I20260619 14:41:37.773622 21724 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea1fec84dee44082917ebf4bff172291 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:41:37.773681 21724 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea1fec84dee44082917ebf4bff172291 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:41:37.773775 21724 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea1fec84dee44082917ebf4bff172291 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:41:37.774837 21724 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea1fec84dee44082917ebf4bff172291 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea1fec84dee44082917ebf4bff172291" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45089 } }
I20260619 14:41:37.775166 21724 leader_election.cc:302] T 00000000000000000000000000000000 P ea1fec84dee44082917ebf4bff172291 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea1fec84dee44082917ebf4bff172291; no voters:
I20260619 14:41:37.775461 21727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:37.760869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.157:59279 (local address 127.0.0.1:45089)
0619 14:41:37.761295 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:37.761314 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:37.761357 (+ 43us) server_negotiation.cc:407] Connection header received
0619 14:41:37.762156 (+ 799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:37.762180 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:37.762627 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:37.763005 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:37.763718 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:37.765252 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:37.766207 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:37.766911 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:37.767086 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:37.769708 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:37.769715 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:37.769733 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:41:37.769763 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:37.772287 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:37.772858 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:37.772865 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:37.772879 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:37.773047 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:37.773372 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:37.773376 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:37.773378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:37.773861 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:37.774080 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:37.774391 (+ 311us) server_negotiation.cc:299] Negotiation successful
0619 14:41:37.774619 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":125,"threads_started":1}
I20260619 14:41:37.775468 21724 leader_election.cc:288] T 00000000000000000000000000000000 P ea1fec84dee44082917ebf4bff172291 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:41:37.775578 21729 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea1fec84dee44082917ebf4bff172291 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:41:37.775770 21729 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea1fec84dee44082917ebf4bff172291 [term 1 LEADER]: Becoming Leader. State: Replica: ea1fec84dee44082917ebf4bff172291, State: Running, Role: LEADER
I20260619 14:41:37.776155 21729 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea1fec84dee44082917ebf4bff172291 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea1fec84dee44082917ebf4bff172291" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45089 } }
I20260619 14:41:37.776767 21724 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea1fec84dee44082917ebf4bff172291 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:41:37.778390 21731 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea1fec84dee44082917ebf4bff172291 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea1fec84dee44082917ebf4bff172291" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea1fec84dee44082917ebf4bff172291" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45089 } } }
I20260619 14:41:37.778542 21731 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea1fec84dee44082917ebf4bff172291 [sys.catalog]: This master's current role is: LEADER
I20260619 14:41:37.779017 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:41:37.779572 21743 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:41:37.779356 21730 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea1fec84dee44082917ebf4bff172291 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea1fec84dee44082917ebf4bff172291. Latest consensus state: current_term: 1 leader_uuid: "ea1fec84dee44082917ebf4bff172291" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea1fec84dee44082917ebf4bff172291" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45089 } } }
I20260619 14:41:37.779667 21730 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea1fec84dee44082917ebf4bff172291 [sys.catalog]: This master's current role is: LEADER
I20260619 14:41:37.782640 21743 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:41:37.788439 21743 catalog_manager.cc:1347] Generated new cluster ID: 761704f099f147928383089d7770635a
I20260619 14:41:37.788566 21743 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:41:37.792323 21727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:37.781029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.157:58283 (local address 127.0.0.1:45089)
0619 14:41:37.781134 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:37.781139 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:37.783535 (+ 2396us) server_negotiation.cc:407] Connection header received
0619 14:41:37.783670 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:37.783673 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:37.783723 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:37.783782 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:37.784456 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:37.785196 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:37.786397 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:37.786730 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:37.786833 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:37.788512 (+ 1679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:37.788517 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:37.788520 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:41:37.788548 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:37.790211 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:37.790776 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:37.790780 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:37.790781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:37.790865 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:37.791281 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:37.791284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:37.791286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:37.791492 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:37.791615 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:37.792067 (+ 452us) server_negotiation.cc:299] Negotiation successful
0619 14:41:37.792164 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 14:41:37.801645 21743 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:41:37.803045 21743 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:41:37.816882 21743 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea1fec84dee44082917ebf4bff172291: Generated new TSK 0
I20260619 14:41:37.817816 21743 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:41:37.821520 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 21656
W20260619 14:41:37.826084 21641 connection.cc:535] client connection to 127.0.0.1:45089 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45089): Connection reset by peer (error 104)
2026-06-19T14:41:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1315 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781879837648036-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:41:37 dist-test-slave-32hj krb5kdc[21755](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:41:37 dist-test-slave-32hj krb5kdc[21755](info): set up 2 sockets
Jun 19 14:41:37 dist-test-slave-32hj krb5kdc[21755](info): commencing operation
krb5kdc: starting...
W20260619 14:41:38.772847 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781879837648036-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781879837648036-392-0/krb5kdc/test-user.keytab.
Jun 19 14:41:38 dist-test-slave-32hj krb5kdc[21755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:41:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:41:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781879837648036-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781879837648036-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:41:38.897043 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781879837648036-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781879837648036-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781879837648036-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45005
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781879837648036-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45005
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:41:39.013098 21768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:41:39.013396 21768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:41:39.013441 21768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:41:39.017529 21768 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:41:39.017604 21768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:41:39.017627 21768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:41:39.017647 21768 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:41:39.017673 21768 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:41:39.023072 21768 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781879837648036-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781879837648036-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45005
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781879837648036-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781879837648036-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:41:39.024091 21768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:41:39.025120 21768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:41:39.033485 21776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:39.033478 21774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:39.033581 21768 server_base.cc:1034] running on GCE node
W20260619 14:41:39.033474 21773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:39.034291 21768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:41:39.035723 21768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:41:39.036904 21768 hybrid_clock.cc:648] HybridClock initialized: now 1781880099036874 us; error 43 us; skew 500 ppm
Jun 19 14:41:39 dist-test-slave-32hj krb5kdc[21755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880099, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:41:39.045591 21768 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:41:39.046792 21768 webserver.cc:460] Webserver started at http://127.0.0.1:36013/ using document root <none> and password file <none>
I20260619 14:41:39.047374 21768 fs_manager.cc:359] Metadata directory not provided
I20260619 14:41:39.047466 21768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:41:39.047676 21768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:41:39.049813 21768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781879837648036-392-0/minicluster-data/master-0/data/instance:
uuid: "c803c4bf0853419ea18460724ae6e598"
format_stamp: "Formatted at 2026-06-19 14:41:39 on dist-test-slave-32hj"
server_key: "7716222a01099e92a8c99cd28e37d4ad"
server_key_iv: "faf429a797e9682fe06a545208d0dba3"
server_key_version: "encryptionkey@0"
I20260619 14:41:39.050364 21768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781879837648036-392-0/minicluster-data/master-0/wal/instance:
uuid: "c803c4bf0853419ea18460724ae6e598"
format_stamp: "Formatted at 2026-06-19 14:41:39 on dist-test-slave-32hj"
server_key: "7716222a01099e92a8c99cd28e37d4ad"
server_key_iv: "faf429a797e9682fe06a545208d0dba3"
server_key_version: "encryptionkey@0"
I20260619 14:41:39.054034 21768 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:41:39.056406 21783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:39.057605 21768 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:41:39.057720 21768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781879837648036-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781879837648036-392-0/minicluster-data/master-0/wal
uuid: "c803c4bf0853419ea18460724ae6e598"
format_stamp: "Formatted at 2026-06-19 14:41:39 on dist-test-slave-32hj"
server_key: "7716222a01099e92a8c99cd28e37d4ad"
server_key_iv: "faf429a797e9682fe06a545208d0dba3"
server_key_version: "encryptionkey@0"
I20260619 14:41:39.057864 21768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781879837648036-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781879837648036-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781879837648036-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:41:39.072000 21768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:41:39.075227 21768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:41:39.075397 21768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:41:39.082433 21768 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45005
I20260619 14:41:39.082448 21835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45005 every 8 connection(s)
I20260619 14:41:39.083384 21768 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:41:39.086118 21836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:41:39.091583 21836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c803c4bf0853419ea18460724ae6e598: Bootstrap starting.
I20260619 14:41:39.093623 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 21768
I20260619 14:41:39.093743 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781879837648036-392-0/minicluster-data/master-0/wal/instance
I20260619 14:41:39.094002 392 external_mini_cluster.cc:1428] Setting key 5d3c08002b23b4b882e3b6f8a41dfe87
I20260619 14:41:39.094146 21836 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c803c4bf0853419ea18460724ae6e598: Neither blocks nor log segments found. Creating new log.
I20260619 14:41:39.095024 21836 log.cc:824] T 00000000000000000000000000000000 P c803c4bf0853419ea18460724ae6e598: Log is configured to *not* fsync() on all Append() calls
I20260619 14:41:39.097280 21836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c803c4bf0853419ea18460724ae6e598: No bootstrap required, opened a new log
I20260619 14:41:39.101579 21836 raft_consensus.cc:348] T 00000000000000000000000000000000 P c803c4bf0853419ea18460724ae6e598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c803c4bf0853419ea18460724ae6e598" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45005 } }
I20260619 14:41:39.101826 21836 raft_consensus.cc:374] T 00000000000000000000000000000000 P c803c4bf0853419ea18460724ae6e598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:41:39.101891 21836 raft_consensus.cc:729] T 00000000000000000000000000000000 P c803c4bf0853419ea18460724ae6e598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c803c4bf0853419ea18460724ae6e598, State: Initialized, Role: FOLLOWER
I20260619 14:41:39.102375 21836 consensus_queue.cc:260] T 00000000000000000000000000000000 P c803c4bf0853419ea18460724ae6e598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c803c4bf0853419ea18460724ae6e598" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45005 } }
I20260619 14:41:39.102532 21836 raft_consensus.cc:388] T 00000000000000000000000000000000 P c803c4bf0853419ea18460724ae6e598 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:41:39.102602 21836 raft_consensus.cc:482] T 00000000000000000000000000000000 P c803c4bf0853419ea18460724ae6e598 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:41:39.102705 21836 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c803c4bf0853419ea18460724ae6e598 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:41:39.103657 21836 raft_consensus.cc:504] T 00000000000000000000000000000000 P c803c4bf0853419ea18460724ae6e598 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c803c4bf0853419ea18460724ae6e598" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45005 } }
I20260619 14:41:39.103806 21839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:39.095119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.157:60933 (local address 127.0.0.1:45005)
0619 14:41:39.095561 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:39.095576 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:39.095611 (+ 35us) server_negotiation.cc:407] Connection header received
0619 14:41:39.096431 (+ 820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:39.096455 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:39.096908 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:39.097271 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:39.097702 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:39.099169 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:39.100108 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:39.100889 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:39.101047 (+ 158us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:39.101918 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:39.101924 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:39.101943 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:41:39.102437 (+ 494us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:39.102505 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:41:39.102536 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:39.102730 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:39.102950 (+ 220us) server_negotiation.cc:299] Negotiation successful
0619 14:41:39.103085 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":131,"threads_started":1}
I20260619 14:41:39.104008 21836 leader_election.cc:302] T 00000000000000000000000000000000 P c803c4bf0853419ea18460724ae6e598 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c803c4bf0853419ea18460724ae6e598; no voters:
I20260619 14:41:39.104367 21836 leader_election.cc:288] T 00000000000000000000000000000000 P c803c4bf0853419ea18460724ae6e598 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:41:39.104472 21841 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c803c4bf0853419ea18460724ae6e598 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:41:39.104677 21841 raft_consensus.cc:686] T 00000000000000000000000000000000 P c803c4bf0853419ea18460724ae6e598 [term 1 LEADER]: Becoming Leader. State: Replica: c803c4bf0853419ea18460724ae6e598, State: Running, Role: LEADER
I20260619 14:41:39.105037 21841 consensus_queue.cc:237] T 00000000000000000000000000000000 P c803c4bf0853419ea18460724ae6e598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c803c4bf0853419ea18460724ae6e598" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45005 } }
I20260619 14:41:39.105572 21836 sys_catalog.cc:564] T 00000000000000000000000000000000 P c803c4bf0853419ea18460724ae6e598 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:41:39.106683 21843 sys_catalog.cc:455] T 00000000000000000000000000000000 P c803c4bf0853419ea18460724ae6e598 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c803c4bf0853419ea18460724ae6e598. Latest consensus state: current_term: 1 leader_uuid: "c803c4bf0853419ea18460724ae6e598" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c803c4bf0853419ea18460724ae6e598" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45005 } } }
I20260619 14:41:39.106858 21843 sys_catalog.cc:458] T 00000000000000000000000000000000 P c803c4bf0853419ea18460724ae6e598 [sys.catalog]: This master's current role is: LEADER
I20260619 14:41:39.107010 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:41:39.107208 21842 sys_catalog.cc:455] T 00000000000000000000000000000000 P c803c4bf0853419ea18460724ae6e598 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c803c4bf0853419ea18460724ae6e598" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c803c4bf0853419ea18460724ae6e598" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45005 } } }
I20260619 14:41:39.107282 21842 sys_catalog.cc:458] T 00000000000000000000000000000000 P c803c4bf0853419ea18460724ae6e598 [sys.catalog]: This master's current role is: LEADER
W20260619 14:41:39.110508 21856 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c803c4bf0853419ea18460724ae6e598: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:41:39.110580 21856 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:41:39.110641 21851 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:41:39.111403 21851 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:41:39.113777 21839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:39.109150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.157:39623 (local address 127.0.0.1:45005)
0619 14:41:39.109244 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:39.109248 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:39.109650 (+ 402us) server_negotiation.cc:407] Connection header received
0619 14:41:39.109757 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:39.109760 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:39.109809 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:39.109892 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:39.110174 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:39.110936 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:39.112015 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:39.112353 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:39.112472 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:39.113348 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:39.113351 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:39.113354 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:41:39.113367 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:39.113391 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:41:39.113401 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:39.113474 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:39.113602 (+ 128us) server_negotiation.cc:299] Negotiation successful
0619 14:41:39.113636 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 14:41:39.117458 21851 catalog_manager.cc:1347] Generated new cluster ID: 93415777507644fc8e6027162ffaa784
I20260619 14:41:39.117540 21851 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:41:39.140811 21851 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:41:39.141835 21851 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:41:39.146809 21851 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c803c4bf0853419ea18460724ae6e598: Generated new TSK 0
I20260619 14:41:39.147557 21851 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:41:39.158282 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 21768
W20260619 14:41:39.163250 21753 connection.cc:535] client connection to 127.0.0.1:45005 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45005): Connection reset by peer (error 104)
2026-06-19T14:41:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1337 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781879837648036-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:41:39 dist-test-slave-32hj krb5kdc[21867](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:41:39 dist-test-slave-32hj krb5kdc[21867](info): set up 2 sockets
Jun 19 14:41:39 dist-test-slave-32hj krb5kdc[21867](info): commencing operation
krb5kdc: starting...
W20260619 14:41:40.120675 392 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781879837648036-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781879837648036-392-0/krb5kdc/test-user.keytab.
Jun 19 14:41:40 dist-test-slave-32hj krb5kdc[21867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:41:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:41:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781879837648036-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781879837648036-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:41:40.239487 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781879837648036-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781879837648036-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781879837648036-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43191
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781879837648036-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43191
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:41:40.350198 21880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:41:40.350485 21880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:41:40.350528 21880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:41:40.354525 21880 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:41:40.354596 21880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:41:40.354615 21880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:41:40.354630 21880 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:41:40.354643 21880 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:41:40.359987 21880 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781879837648036-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781879837648036-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43191
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781879837648036-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781879837648036-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:41:40.361121 21880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:41:40.362133 21880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:41:40.370748 21880 server_base.cc:1034] running on GCE node
W20260619 14:41:40.370813 21888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:40.370949 21886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:40.370817 21885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:40.371551 21880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:41:40.372747 21880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:41:40.373943 21880 hybrid_clock.cc:648] HybridClock initialized: now 1781880100373926 us; error 37 us; skew 500 ppm
Jun 19 14:41:40 dist-test-slave-32hj krb5kdc[21867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880100, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:41:40.376999 21880 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:41:40.377924 21880 webserver.cc:460] Webserver started at http://127.0.0.1:38955/ using document root <none> and password file <none>
I20260619 14:41:40.378582 21880 fs_manager.cc:359] Metadata directory not provided
I20260619 14:41:40.378660 21880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:41:40.378954 21880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:41:40.381039 21880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781879837648036-392-0/minicluster-data/master-0/data/instance:
uuid: "701acd6bcfd741db9c9e2863a3c79239"
format_stamp: "Formatted at 2026-06-19 14:41:40 on dist-test-slave-32hj"
server_key: "d4eb6d5cf16c78f1b170c45a228a2a0b"
server_key_iv: "b80f8b2e4da35570a9f9fc98770f3277"
server_key_version: "encryptionkey@0"
I20260619 14:41:40.381568 21880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781879837648036-392-0/minicluster-data/master-0/wal/instance:
uuid: "701acd6bcfd741db9c9e2863a3c79239"
format_stamp: "Formatted at 2026-06-19 14:41:40 on dist-test-slave-32hj"
server_key: "d4eb6d5cf16c78f1b170c45a228a2a0b"
server_key_iv: "b80f8b2e4da35570a9f9fc98770f3277"
server_key_version: "encryptionkey@0"
I20260619 14:41:40.385320 21880 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:41:40.387734 21895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:40.389029 21880 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:41:40.389149 21880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781879837648036-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781879837648036-392-0/minicluster-data/master-0/wal
uuid: "701acd6bcfd741db9c9e2863a3c79239"
format_stamp: "Formatted at 2026-06-19 14:41:40 on dist-test-slave-32hj"
server_key: "d4eb6d5cf16c78f1b170c45a228a2a0b"
server_key_iv: "b80f8b2e4da35570a9f9fc98770f3277"
server_key_version: "encryptionkey@0"
I20260619 14:41:40.389250 21880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781879837648036-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781879837648036-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781879837648036-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:41:40.393802 21880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:41:40.397166 21880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:41:40.397339 21880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:41:40.403959 21880 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43191
I20260619 14:41:40.404012 21947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43191 every 8 connection(s)
I20260619 14:41:40.404968 21880 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:41:40.405316 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 21880
I20260619 14:41:40.405467 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781879837648036-392-0/minicluster-data/master-0/wal/instance
I20260619 14:41:40.405771 392 external_mini_cluster.cc:1428] Setting key fec14776db4652db9b5aee7008a00021
I20260619 14:41:40.408304 21948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:41:40.411151 21951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:40.407333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.157:43633 (local address 127.0.0.1:43191)
0619 14:41:40.407785 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:40.407800 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:40.407837 (+ 37us) server_negotiation.cc:407] Connection header received
0619 14:41:40.408511 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:40.408529 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:40.408864 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:40.409153 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:40.409978 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:40.409983 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:40.409994 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:41:40.410005 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:40.410057 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:41:40.410076 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:40.410183 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:40.410283 (+ 100us) server_negotiation.cc:299] Negotiation successful
0619 14:41:40.410438 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":134,"threads_started":1}
I20260619 14:41:40.414122 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:41:40.414631 21948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 701acd6bcfd741db9c9e2863a3c79239: Bootstrap starting.
I20260619 14:41:40.417270 21948 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 701acd6bcfd741db9c9e2863a3c79239: Neither blocks nor log segments found. Creating new log.
I20260619 14:41:40.418077 21951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:40.416430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.157:59303 (local address 127.0.0.1:43191)
0619 14:41:40.416554 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:40.416558 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:40.416779 (+ 221us) server_negotiation.cc:407] Connection header received
0619 14:41:40.416893 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:40.416896 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:40.416934 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:40.417017 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:40.417673 (+ 656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:40.417676 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:40.417677 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:41:40.417688 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:40.417714 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:41:40.417725 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:40.417786 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:40.417914 (+ 128us) server_negotiation.cc:299] Negotiation successful
0619 14:41:40.417952 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 14:41:40.418105 21948 log.cc:824] T 00000000000000000000000000000000 P 701acd6bcfd741db9c9e2863a3c79239: Log is configured to *not* fsync() on all Append() calls
I20260619 14:41:40.420135 21948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 701acd6bcfd741db9c9e2863a3c79239: No bootstrap required, opened a new log
I20260619 14:41:40.423673 21948 raft_consensus.cc:348] T 00000000000000000000000000000000 P 701acd6bcfd741db9c9e2863a3c79239 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "701acd6bcfd741db9c9e2863a3c79239" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43191 } }
I20260619 14:41:40.423911 21948 raft_consensus.cc:374] T 00000000000000000000000000000000 P 701acd6bcfd741db9c9e2863a3c79239 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:41:40.423959 21948 raft_consensus.cc:729] T 00000000000000000000000000000000 P 701acd6bcfd741db9c9e2863a3c79239 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 701acd6bcfd741db9c9e2863a3c79239, State: Initialized, Role: FOLLOWER
I20260619 14:41:40.424372 21948 consensus_queue.cc:260] T 00000000000000000000000000000000 P 701acd6bcfd741db9c9e2863a3c79239 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "701acd6bcfd741db9c9e2863a3c79239" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43191 } }
I20260619 14:41:40.424500 21948 raft_consensus.cc:388] T 00000000000000000000000000000000 P 701acd6bcfd741db9c9e2863a3c79239 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:41:40.424562 21948 raft_consensus.cc:482] T 00000000000000000000000000000000 P 701acd6bcfd741db9c9e2863a3c79239 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:41:40.424655 21948 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 701acd6bcfd741db9c9e2863a3c79239 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:41:40.425576 21948 raft_consensus.cc:504] T 00000000000000000000000000000000 P 701acd6bcfd741db9c9e2863a3c79239 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "701acd6bcfd741db9c9e2863a3c79239" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43191 } }
I20260619 14:41:40.425922 21948 leader_election.cc:302] T 00000000000000000000000000000000 P 701acd6bcfd741db9c9e2863a3c79239 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 701acd6bcfd741db9c9e2863a3c79239; no voters:
I20260619 14:41:40.426214 21948 leader_election.cc:288] T 00000000000000000000000000000000 P 701acd6bcfd741db9c9e2863a3c79239 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:41:40.426333 21958 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 701acd6bcfd741db9c9e2863a3c79239 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:41:40.426569 21958 raft_consensus.cc:686] T 00000000000000000000000000000000 P 701acd6bcfd741db9c9e2863a3c79239 [term 1 LEADER]: Becoming Leader. State: Replica: 701acd6bcfd741db9c9e2863a3c79239, State: Running, Role: LEADER
I20260619 14:41:40.426878 21958 consensus_queue.cc:237] T 00000000000000000000000000000000 P 701acd6bcfd741db9c9e2863a3c79239 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "701acd6bcfd741db9c9e2863a3c79239" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43191 } }
I20260619 14:41:40.427455 21948 sys_catalog.cc:564] T 00000000000000000000000000000000 P 701acd6bcfd741db9c9e2863a3c79239 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:41:40.428566 21959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 701acd6bcfd741db9c9e2863a3c79239 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "701acd6bcfd741db9c9e2863a3c79239" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "701acd6bcfd741db9c9e2863a3c79239" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43191 } } }
I20260619 14:41:40.428764 21959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 701acd6bcfd741db9c9e2863a3c79239 [sys.catalog]: This master's current role is: LEADER
I20260619 14:41:40.428591 21960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 701acd6bcfd741db9c9e2863a3c79239 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 701acd6bcfd741db9c9e2863a3c79239. Latest consensus state: current_term: 1 leader_uuid: "701acd6bcfd741db9c9e2863a3c79239" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "701acd6bcfd741db9c9e2863a3c79239" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43191 } } }
I20260619 14:41:40.428869 21960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 701acd6bcfd741db9c9e2863a3c79239 [sys.catalog]: This master's current role is: LEADER
W20260619 14:41:40.431993 21973 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 701acd6bcfd741db9c9e2863a3c79239: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:41:40.432077 21973 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:41:40.432185 21967 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:41:40.433151 21967 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:41:40.438619 21967 catalog_manager.cc:1347] Generated new cluster ID: ec2a54bf5b0f4783a42d2121896c04a6
I20260619 14:41:40.438695 21967 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:41:40.451941 21967 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:41:40.452284 21967 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:41:40.458249 21967 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 701acd6bcfd741db9c9e2863a3c79239: Generated new TSK 0
I20260619 14:41:40.459013 21967 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:41:40.462226 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 21880
2026-06-19T14:41:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1303 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781879837648036-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:41:40 dist-test-slave-32hj krb5kdc[21979](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:41:40 dist-test-slave-32hj krb5kdc[21979](info): set up 2 sockets
Jun 19 14:41:40 dist-test-slave-32hj krb5kdc[21979](info): commencing operation
krb5kdc: starting...
W20260619 14:41:41.420686 392 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781879837648036-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781879837648036-392-0/krb5kdc/test-user.keytab.
Jun 19 14:41:41 dist-test-slave-32hj krb5kdc[21979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:41:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:41:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:41:41.539455 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781879837648036-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781879837648036-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781879837648036-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40939
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40939
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:41:41.650141 21992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:41:41.650430 21992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:41:41.650472 21992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:41:41.654590 21992 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:41:41.654690 21992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:41:41.654719 21992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:41:41.654743 21992 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:41:41.654765 21992 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:41:41.660202 21992 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781879837648036-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781879837648036-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40939
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781879837648036-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:41:41.661306 21992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:41:41.662290 21992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:41:41.670928 22000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:41.671027 21992 server_base.cc:1034] running on GCE node
W20260619 14:41:41.670912 21997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:41.670912 21998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:41.671828 21992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:41:41.672989 21992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:41:41.674171 21992 hybrid_clock.cc:648] HybridClock initialized: now 1781880101674134 us; error 51 us; skew 500 ppm
Jun 19 14:41:41 dist-test-slave-32hj krb5kdc[21979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880101, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:41:41.676959 21992 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:41:41.677865 21992 webserver.cc:460] Webserver started at http://127.0.98.62:44435/ using document root <none> and password file <none>
I20260619 14:41:41.678412 21992 fs_manager.cc:359] Metadata directory not provided
I20260619 14:41:41.678479 21992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:41:41.678670 21992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:41:41.680796 21992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781879837648036-392-0/minicluster-data/master-0/data/instance:
uuid: "42e08feff80b4ac5a06753d0a1b103fc"
format_stamp: "Formatted at 2026-06-19 14:41:41 on dist-test-slave-32hj"
server_key: "83f90ed04cf9f503638cab1c19567548"
server_key_iv: "1708b843b5e6facce7fbb3afd57e3947"
server_key_version: "encryptionkey@0"
I20260619 14:41:41.681386 21992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781879837648036-392-0/minicluster-data/master-0/wal/instance:
uuid: "42e08feff80b4ac5a06753d0a1b103fc"
format_stamp: "Formatted at 2026-06-19 14:41:41 on dist-test-slave-32hj"
server_key: "83f90ed04cf9f503638cab1c19567548"
server_key_iv: "1708b843b5e6facce7fbb3afd57e3947"
server_key_version: "encryptionkey@0"
I20260619 14:41:41.685213 21992 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.002s
I20260619 14:41:41.687733 22007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:41.689013 21992 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:41:41.689134 21992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781879837648036-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781879837648036-392-0/minicluster-data/master-0/wal
uuid: "42e08feff80b4ac5a06753d0a1b103fc"
format_stamp: "Formatted at 2026-06-19 14:41:41 on dist-test-slave-32hj"
server_key: "83f90ed04cf9f503638cab1c19567548"
server_key_iv: "1708b843b5e6facce7fbb3afd57e3947"
server_key_version: "encryptionkey@0"
I20260619 14:41:41.689240 21992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781879837648036-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781879837648036-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781879837648036-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:41:41.712077 21992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:41:41.715440 21992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:41:41.715644 21992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:41:41.722622 21992 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40939
I20260619 14:41:41.722640 22059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40939 every 8 connection(s)
I20260619 14:41:41.723606 21992 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:41:41.725927 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 21992
I20260619 14:41:41.726054 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781879837648036-392-0/minicluster-data/master-0/wal/instance
I20260619 14:41:41.726338 392 external_mini_cluster.cc:1428] Setting key a9d324fa66d3df2949a68136337c5f62
I20260619 14:41:41.726707 22060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:41:41.733174 22060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42e08feff80b4ac5a06753d0a1b103fc: Bootstrap starting.
Jun 19 14:41:41 dist-test-slave-32hj krb5kdc[21979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:41:41.735628 22060 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42e08feff80b4ac5a06753d0a1b103fc: Neither blocks nor log segments found. Creating new log.
I20260619 14:41:41.736354 22060 log.cc:824] T 00000000000000000000000000000000 P 42e08feff80b4ac5a06753d0a1b103fc: Log is configured to *not* fsync() on all Append() calls
I20260619 14:41:41.738399 22060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42e08feff80b4ac5a06753d0a1b103fc: No bootstrap required, opened a new log
I20260619 14:41:41.740881 22063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:41.727815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56541 (local address 127.0.98.62:40939)
0619 14:41:41.728275 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:41.728291 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:41.728329 (+ 38us) server_negotiation.cc:407] Connection header received
0619 14:41:41.729069 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:41.729088 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:41.729452 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:41.729770 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:41.730574 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:41.731584 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:41.732434 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:41.732959 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:41.733106 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:41.735731 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:41.735738 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:41.735758 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:41:41.735789 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:41.737566 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:41.738069 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:41.738074 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:41.738079 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:41.738194 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:41.738566 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:41.738569 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:41.738571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:41.739096 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:41.739308 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:41.739614 (+ 306us) server_negotiation.cc:299] Negotiation successful
0619 14:41:41.739844 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":154,"threads_started":1}
I20260619 14:41:41.742802 22060 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42e08feff80b4ac5a06753d0a1b103fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42e08feff80b4ac5a06753d0a1b103fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40939 } }
I20260619 14:41:41.743073 22060 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42e08feff80b4ac5a06753d0a1b103fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:41:41.743135 22060 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42e08feff80b4ac5a06753d0a1b103fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42e08feff80b4ac5a06753d0a1b103fc, State: Initialized, Role: FOLLOWER
I20260619 14:41:41.743578 22060 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42e08feff80b4ac5a06753d0a1b103fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42e08feff80b4ac5a06753d0a1b103fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40939 } }
I20260619 14:41:41.743737 22060 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42e08feff80b4ac5a06753d0a1b103fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:41:41.743857 22060 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42e08feff80b4ac5a06753d0a1b103fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:41:41.743963 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:41:41.743954 22060 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42e08feff80b4ac5a06753d0a1b103fc [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:41:41.745190 22060 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42e08feff80b4ac5a06753d0a1b103fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42e08feff80b4ac5a06753d0a1b103fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40939 } }
I20260619 14:41:41.745579 22060 leader_election.cc:302] T 00000000000000000000000000000000 P 42e08feff80b4ac5a06753d0a1b103fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42e08feff80b4ac5a06753d0a1b103fc; no voters:
I20260619 14:41:41.745944 22060 leader_election.cc:288] T 00000000000000000000000000000000 P 42e08feff80b4ac5a06753d0a1b103fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:41:41.746011 22069 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42e08feff80b4ac5a06753d0a1b103fc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:41:41.746232 22069 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42e08feff80b4ac5a06753d0a1b103fc [term 1 LEADER]: Becoming Leader. State: Replica: 42e08feff80b4ac5a06753d0a1b103fc, State: Running, Role: LEADER
I20260619 14:41:41.746639 22069 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42e08feff80b4ac5a06753d0a1b103fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42e08feff80b4ac5a06753d0a1b103fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40939 } }
I20260619 14:41:41.747256 22060 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42e08feff80b4ac5a06753d0a1b103fc [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:41:41.747802 22071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42e08feff80b4ac5a06753d0a1b103fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42e08feff80b4ac5a06753d0a1b103fc. Latest consensus state: current_term: 1 leader_uuid: "42e08feff80b4ac5a06753d0a1b103fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42e08feff80b4ac5a06753d0a1b103fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40939 } } }
I20260619 14:41:41.747946 22071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42e08feff80b4ac5a06753d0a1b103fc [sys.catalog]: This master's current role is: LEADER
I20260619 14:41:41.748229 22070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42e08feff80b4ac5a06753d0a1b103fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42e08feff80b4ac5a06753d0a1b103fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42e08feff80b4ac5a06753d0a1b103fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40939 } } }
I20260619 14:41:41.748317 22070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42e08feff80b4ac5a06753d0a1b103fc [sys.catalog]: This master's current role is: LEADER
I20260619 14:41:41.748639 22076 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:41:41.752199 22076 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:41:41.755681 22063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:41.746173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51509 (local address 127.0.98.62:40939)
0619 14:41:41.746285 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:41.746290 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:41.746713 (+ 423us) server_negotiation.cc:407] Connection header received
0619 14:41:41.746884 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:41.746888 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:41.746945 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:41.747028 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:41.747978 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:41.748790 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:41.749945 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:41.750298 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:41.750377 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:41.752104 (+ 1727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:41.752108 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:41.752110 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:41:41.752136 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:41.753650 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:41.754101 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:41.754105 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:41.754107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:41.754185 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:41.754507 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:41.754510 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:41.754512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:41.754686 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:41.754805 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:41.755443 (+ 638us) server_negotiation.cc:299] Negotiation successful
0619 14:41:41.755537 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 14:41:41.759021 22076 catalog_manager.cc:1347] Generated new cluster ID: 9dc95d11845d4e6383e09c41e9ce1d09
I20260619 14:41:41.759083 22076 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:41:41.766093 22076 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:41:41.766974 22076 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:41:41.771941 22076 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42e08feff80b4ac5a06753d0a1b103fc: Generated new TSK 0
I20260619 14:41:41.772603 22076 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:41:41.779429 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 21992
W20260619 14:41:41.783995 21977 connection.cc:535] client connection to 127.0.98.62:40939 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:40939): Connection reset by peer (error 104)
2026-06-19T14:41:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1317 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781879837648036-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:41:41 dist-test-slave-32hj krb5kdc[22091](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:41:41 dist-test-slave-32hj krb5kdc[22091](info): set up 2 sockets
Jun 19 14:41:41 dist-test-slave-32hj krb5kdc[22091](info): commencing operation
krb5kdc: starting...
W20260619 14:41:42.740617 392 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781879837648036-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781879837648036-392-0/krb5kdc/test-user.keytab.
Jun 19 14:41:42 dist-test-slave-32hj krb5kdc[22091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:41:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:41:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:41:42.853072 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781879837648036-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781879837648036-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781879837648036-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41029
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41029
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:41:42.960973 22104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:41:42.961268 22104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:41:42.961309 22104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:41:42.965760 22104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:41:42.965842 22104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:41:42.965862 22104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:41:42.965878 22104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:41:42.965891 22104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:41:42.971397 22104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781879837648036-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781879837648036-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41029
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781879837648036-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:41:42.972507 22104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:41:42.973604 22104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:41:42.982322 22112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:42.982306 22110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:42.982270 22109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:42.982299 22104 server_base.cc:1034] running on GCE node
I20260619 14:41:42.983070 22104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:41:42.984241 22104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:41:42.985397 22104 hybrid_clock.cc:648] HybridClock initialized: now 1781880102985382 us; error 32 us; skew 500 ppm
Jun 19 14:41:42 dist-test-slave-32hj krb5kdc[22091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880102, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:41:42.988106 22104 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:41:42.989076 22104 webserver.cc:460] Webserver started at http://127.0.98.62:44099/ using document root <none> and password file <none>
I20260619 14:41:42.989624 22104 fs_manager.cc:359] Metadata directory not provided
I20260619 14:41:42.989691 22104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:41:42.989888 22104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:41:42.991959 22104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781879837648036-392-0/minicluster-data/master-0/data/instance:
uuid: "d410aaf155664f56bd5ee7913666ff09"
format_stamp: "Formatted at 2026-06-19 14:41:42 on dist-test-slave-32hj"
server_key: "dad4ad1fdad99894510622c851d62f8b"
server_key_iv: "d1178a4b35911cbaedf511a02050837a"
server_key_version: "encryptionkey@0"
I20260619 14:41:42.992512 22104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781879837648036-392-0/minicluster-data/master-0/wal/instance:
uuid: "d410aaf155664f56bd5ee7913666ff09"
format_stamp: "Formatted at 2026-06-19 14:41:42 on dist-test-slave-32hj"
server_key: "dad4ad1fdad99894510622c851d62f8b"
server_key_iv: "d1178a4b35911cbaedf511a02050837a"
server_key_version: "encryptionkey@0"
I20260619 14:41:42.997485 22104 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260619 14:41:43.000547 22119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:43.002046 22104 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:41:43.002172 22104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781879837648036-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781879837648036-392-0/minicluster-data/master-0/wal
uuid: "d410aaf155664f56bd5ee7913666ff09"
format_stamp: "Formatted at 2026-06-19 14:41:42 on dist-test-slave-32hj"
server_key: "dad4ad1fdad99894510622c851d62f8b"
server_key_iv: "d1178a4b35911cbaedf511a02050837a"
server_key_version: "encryptionkey@0"
I20260619 14:41:43.002282 22104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781879837648036-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781879837648036-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781879837648036-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:41:43.021598 22104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:41:43.024883 22104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:41:43.025043 22104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:41:43.031627 22104 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41029
I20260619 14:41:43.031701 22171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41029 every 8 connection(s)
I20260619 14:41:43.032563 22104 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:41:43.035356 22172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:41:43.038952 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 22104
I20260619 14:41:43.039062 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781879837648036-392-0/minicluster-data/master-0/wal/instance
I20260619 14:41:43.039322 392 external_mini_cluster.cc:1428] Setting key f0fe8735f0f3b2be7b2c08e27bfc05a1
I20260619 14:41:43.041576 22172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d410aaf155664f56bd5ee7913666ff09: Bootstrap starting.
I20260619 14:41:43.043797 22172 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d410aaf155664f56bd5ee7913666ff09: Neither blocks nor log segments found. Creating new log.
I20260619 14:41:43.044519 22172 log.cc:824] T 00000000000000000000000000000000 P d410aaf155664f56bd5ee7913666ff09: Log is configured to *not* fsync() on all Append() calls
I20260619 14:41:43.046828 22172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d410aaf155664f56bd5ee7913666ff09: No bootstrap required, opened a new log
I20260619 14:41:43.049324 22175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:43.040800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53423 (local address 127.0.98.62:41029)
0619 14:41:43.041253 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:43.041268 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:43.041318 (+ 50us) server_negotiation.cc:407] Connection header received
0619 14:41:43.041985 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:43.042004 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:43.042356 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:43.042662 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:43.043098 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:43.044127 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:43.045256 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:43.045963 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:43.046134 (+ 171us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:43.047000 (+ 866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:43.047006 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:43.047024 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:41:43.047550 (+ 526us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:43.047617 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:41:43.047647 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:43.047793 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:43.047943 (+ 150us) server_negotiation.cc:299] Negotiation successful
0619 14:41:43.048082 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":148,"threads_started":1}
I20260619 14:41:43.050657 22172 raft_consensus.cc:348] T 00000000000000000000000000000000 P d410aaf155664f56bd5ee7913666ff09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d410aaf155664f56bd5ee7913666ff09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41029 } }
I20260619 14:41:43.050931 22172 raft_consensus.cc:374] T 00000000000000000000000000000000 P d410aaf155664f56bd5ee7913666ff09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:41:43.050987 22172 raft_consensus.cc:729] T 00000000000000000000000000000000 P d410aaf155664f56bd5ee7913666ff09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d410aaf155664f56bd5ee7913666ff09, State: Initialized, Role: FOLLOWER
I20260619 14:41:43.051440 22172 consensus_queue.cc:260] T 00000000000000000000000000000000 P d410aaf155664f56bd5ee7913666ff09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d410aaf155664f56bd5ee7913666ff09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41029 } }
I20260619 14:41:43.051573 22172 raft_consensus.cc:388] T 00000000000000000000000000000000 P d410aaf155664f56bd5ee7913666ff09 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:41:43.051632 22172 raft_consensus.cc:482] T 00000000000000000000000000000000 P d410aaf155664f56bd5ee7913666ff09 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:41:43.051729 22172 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d410aaf155664f56bd5ee7913666ff09 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:41:43.052492 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:41:43.052832 22172 raft_consensus.cc:504] T 00000000000000000000000000000000 P d410aaf155664f56bd5ee7913666ff09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d410aaf155664f56bd5ee7913666ff09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41029 } }
I20260619 14:41:43.053156 22172 leader_election.cc:302] T 00000000000000000000000000000000 P d410aaf155664f56bd5ee7913666ff09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d410aaf155664f56bd5ee7913666ff09; no voters:
I20260619 14:41:43.053469 22172 leader_election.cc:288] T 00000000000000000000000000000000 P d410aaf155664f56bd5ee7913666ff09 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:41:43.053555 22177 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d410aaf155664f56bd5ee7913666ff09 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:41:43.053710 22177 raft_consensus.cc:686] T 00000000000000000000000000000000 P d410aaf155664f56bd5ee7913666ff09 [term 1 LEADER]: Becoming Leader. State: Replica: d410aaf155664f56bd5ee7913666ff09, State: Running, Role: LEADER
I20260619 14:41:43.054073 22177 consensus_queue.cc:237] T 00000000000000000000000000000000 P d410aaf155664f56bd5ee7913666ff09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d410aaf155664f56bd5ee7913666ff09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41029 } }
I20260619 14:41:43.054795 22172 sys_catalog.cc:564] T 00000000000000000000000000000000 P d410aaf155664f56bd5ee7913666ff09 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:41:43.055797 22178 sys_catalog.cc:455] T 00000000000000000000000000000000 P d410aaf155664f56bd5ee7913666ff09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d410aaf155664f56bd5ee7913666ff09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d410aaf155664f56bd5ee7913666ff09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41029 } } }
I20260619 14:41:43.055917 22178 sys_catalog.cc:458] T 00000000000000000000000000000000 P d410aaf155664f56bd5ee7913666ff09 [sys.catalog]: This master's current role is: LEADER
I20260619 14:41:43.056142 22178 sys_catalog.cc:455] T 00000000000000000000000000000000 P d410aaf155664f56bd5ee7913666ff09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d410aaf155664f56bd5ee7913666ff09. Latest consensus state: current_term: 1 leader_uuid: "d410aaf155664f56bd5ee7913666ff09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d410aaf155664f56bd5ee7913666ff09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41029 } } }
I20260619 14:41:43.056212 22178 sys_catalog.cc:458] T 00000000000000000000000000000000 P d410aaf155664f56bd5ee7913666ff09 [sys.catalog]: This master's current role is: LEADER
I20260619 14:41:43.056588 22191 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:41:43.059414 22175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:43.054689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35547 (local address 127.0.98.62:41029)
0619 14:41:43.054822 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:43.054827 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:43.055007 (+ 180us) server_negotiation.cc:407] Connection header received
0619 14:41:43.055166 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:43.055169 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:43.055218 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:43.055287 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:43.055738 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:43.056493 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:43.057651 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:43.057898 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:43.057991 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:43.058900 (+ 909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:43.058904 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:43.058906 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:41:43.058920 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:43.058948 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:41:43.058962 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:43.059049 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:43.059246 (+ 197us) server_negotiation.cc:299] Negotiation successful
0619 14:41:43.059285 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 14:41:43.059929 22191 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:41:43.065388 22191 catalog_manager.cc:1347] Generated new cluster ID: d495177708544322ab24e6e6ea3e97e6
I20260619 14:41:43.065459 22191 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:41:43.099074 22191 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:41:43.100404 22191 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:41:43.110077 22191 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d410aaf155664f56bd5ee7913666ff09: Generated new TSK 0
I20260619 14:41:43.110981 22191 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:41:43.118569 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 22104
W20260619 14:41:43.123236 22089 connection.cc:535] client connection to 127.0.98.62:41029 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:41029): Connection reset by peer (error 104)
2026-06-19T14:41:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1338 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781879837648036-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:41:43 dist-test-slave-32hj krb5kdc[22203](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:41:43 dist-test-slave-32hj krb5kdc[22203](info): set up 2 sockets
Jun 19 14:41:43 dist-test-slave-32hj krb5kdc[22203](info): commencing operation
krb5kdc: starting...
W20260619 14:41:44.068868 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781879837648036-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781879837648036-392-0/krb5kdc/test-user.keytab.
Jun 19 14:41:44 dist-test-slave-32hj krb5kdc[22203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880104, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:41:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:41:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:41:44.186132 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781879837648036-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781879837648036-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781879837648036-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34863
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34863
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:41:44.296810 22216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:41:44.297106 22216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:41:44.297150 22216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:41:44.301183 22216 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:41:44.301246 22216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:41:44.301265 22216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:41:44.301281 22216 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:41:44.301295 22216 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:41:44.306641 22216 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781879837648036-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781879837648036-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34863
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22216
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781879837648036-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:41:44.307730 22216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:41:44.308748 22216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:41:44.317366 22222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:44.317371 22224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:44.317369 22221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:44.317454 22216 server_base.cc:1034] running on GCE node
I20260619 14:41:44.318240 22216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:41:44.319555 22216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:41:44.320765 22216 hybrid_clock.cc:648] HybridClock initialized: now 1781880104320734 us; error 46 us; skew 500 ppm
Jun 19 14:41:44 dist-test-slave-32hj krb5kdc[22203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880104, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:41:44.328101 22216 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:41:44.329229 22216 webserver.cc:460] Webserver started at http://127.0.98.62:40153/ using document root <none> and password file <none>
I20260619 14:41:44.329820 22216 fs_manager.cc:359] Metadata directory not provided
I20260619 14:41:44.329891 22216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:41:44.330104 22216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:41:44.332504 22216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781879837648036-392-0/minicluster-data/master-0/data/instance:
uuid: "d7b37f02ae494a2dab4675da2e45ccf4"
format_stamp: "Formatted at 2026-06-19 14:41:44 on dist-test-slave-32hj"
server_key: "3d38b325a233f70d3ccb8e7a89323951"
server_key_iv: "cd758ceda41690813fe6e75550d9acb6"
server_key_version: "encryptionkey@0"
I20260619 14:41:44.333148 22216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781879837648036-392-0/minicluster-data/master-0/wal/instance:
uuid: "d7b37f02ae494a2dab4675da2e45ccf4"
format_stamp: "Formatted at 2026-06-19 14:41:44 on dist-test-slave-32hj"
server_key: "3d38b325a233f70d3ccb8e7a89323951"
server_key_iv: "cd758ceda41690813fe6e75550d9acb6"
server_key_version: "encryptionkey@0"
I20260619 14:41:44.337087 22216 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:41:44.339653 22231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:44.341017 22216 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:41:44.341147 22216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781879837648036-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781879837648036-392-0/minicluster-data/master-0/wal
uuid: "d7b37f02ae494a2dab4675da2e45ccf4"
format_stamp: "Formatted at 2026-06-19 14:41:44 on dist-test-slave-32hj"
server_key: "3d38b325a233f70d3ccb8e7a89323951"
server_key_iv: "cd758ceda41690813fe6e75550d9acb6"
server_key_version: "encryptionkey@0"
I20260619 14:41:44.341260 22216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781879837648036-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781879837648036-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781879837648036-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:41:44.345813 22216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:41:44.349119 22216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:41:44.349308 22216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:41:44.356112 22216 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34863
I20260619 14:41:44.356156 22283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34863 every 8 connection(s)
I20260619 14:41:44.357096 22216 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:41:44.359926 22284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:41:44.362366 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 22216
I20260619 14:41:44.362511 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781879837648036-392-0/minicluster-data/master-0/wal/instance
I20260619 14:41:44.362802 392 external_mini_cluster.cc:1428] Setting key 1712990f8819dd2716e1a450a318137b
I20260619 14:41:44.366241 22284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7b37f02ae494a2dab4675da2e45ccf4: Bootstrap starting.
I20260619 14:41:44.368086 22287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:44.364301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34297 (local address 127.0.98.62:34863)
0619 14:41:44.364811 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:44.364828 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:44.364866 (+ 38us) server_negotiation.cc:407] Connection header received
0619 14:41:44.365521 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:44.365539 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:44.365851 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:44.366172 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:44.366885 (+ 713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:44.366889 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:44.366902 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:41:44.366911 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:44.366963 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:41:44.366981 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:44.367109 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:44.367181 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 14:41:44.367360 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":162,"threads_started":1}
I20260619 14:41:44.368942 22284 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7b37f02ae494a2dab4675da2e45ccf4: Neither blocks nor log segments found. Creating new log.
I20260619 14:41:44.369710 22284 log.cc:824] T 00000000000000000000000000000000 P d7b37f02ae494a2dab4675da2e45ccf4: Log is configured to *not* fsync() on all Append() calls
I20260619 14:41:44.370949 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:41:44.371754 22284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7b37f02ae494a2dab4675da2e45ccf4: No bootstrap required, opened a new log
I20260619 14:41:44.374329 22287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:44.372885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52187 (local address 127.0.98.62:34863)
0619 14:41:44.373008 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:44.373013 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:44.373157 (+ 144us) server_negotiation.cc:407] Connection header received
0619 14:41:44.373248 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:44.373251 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:44.373290 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:44.373373 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:44.373980 (+ 607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:44.373984 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:44.373987 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:41:44.373998 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:44.374025 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:41:44.374037 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:44.374127 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:44.374158 (+ 31us) server_negotiation.cc:299] Negotiation successful
0619 14:41:44.374195 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 14:41:44.375771 22284 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7b37f02ae494a2dab4675da2e45ccf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7b37f02ae494a2dab4675da2e45ccf4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34863 } }
I20260619 14:41:44.376021 22284 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7b37f02ae494a2dab4675da2e45ccf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:41:44.376057 22284 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7b37f02ae494a2dab4675da2e45ccf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7b37f02ae494a2dab4675da2e45ccf4, State: Initialized, Role: FOLLOWER
I20260619 14:41:44.376470 22284 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7b37f02ae494a2dab4675da2e45ccf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7b37f02ae494a2dab4675da2e45ccf4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34863 } }
I20260619 14:41:44.376595 22284 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7b37f02ae494a2dab4675da2e45ccf4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:41:44.376657 22284 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7b37f02ae494a2dab4675da2e45ccf4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:41:44.376766 22284 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7b37f02ae494a2dab4675da2e45ccf4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:41:44.377734 22284 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7b37f02ae494a2dab4675da2e45ccf4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7b37f02ae494a2dab4675da2e45ccf4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34863 } }
I20260619 14:41:44.378074 22284 leader_election.cc:302] T 00000000000000000000000000000000 P d7b37f02ae494a2dab4675da2e45ccf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7b37f02ae494a2dab4675da2e45ccf4; no voters:
I20260619 14:41:44.378360 22284 leader_election.cc:288] T 00000000000000000000000000000000 P d7b37f02ae494a2dab4675da2e45ccf4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:41:44.378489 22294 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7b37f02ae494a2dab4675da2e45ccf4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:41:44.378713 22294 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7b37f02ae494a2dab4675da2e45ccf4 [term 1 LEADER]: Becoming Leader. State: Replica: d7b37f02ae494a2dab4675da2e45ccf4, State: Running, Role: LEADER
I20260619 14:41:44.379104 22294 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7b37f02ae494a2dab4675da2e45ccf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7b37f02ae494a2dab4675da2e45ccf4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34863 } }
I20260619 14:41:44.379631 22284 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7b37f02ae494a2dab4675da2e45ccf4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:41:44.380770 22295 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7b37f02ae494a2dab4675da2e45ccf4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7b37f02ae494a2dab4675da2e45ccf4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7b37f02ae494a2dab4675da2e45ccf4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34863 } } }
I20260619 14:41:44.380897 22295 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7b37f02ae494a2dab4675da2e45ccf4 [sys.catalog]: This master's current role is: LEADER
I20260619 14:41:44.381289 22304 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:41:44.381258 22296 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7b37f02ae494a2dab4675da2e45ccf4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7b37f02ae494a2dab4675da2e45ccf4. Latest consensus state: current_term: 1 leader_uuid: "d7b37f02ae494a2dab4675da2e45ccf4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7b37f02ae494a2dab4675da2e45ccf4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34863 } } }
I20260619 14:41:44.381433 22296 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7b37f02ae494a2dab4675da2e45ccf4 [sys.catalog]: This master's current role is: LEADER
I20260619 14:41:44.384353 22304 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:41:44.389784 22304 catalog_manager.cc:1347] Generated new cluster ID: ac189d5e851e4ca2a722ec3ab66642cf
I20260619 14:41:44.389868 22304 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:41:44.402230 22304 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:41:44.402581 22304 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:41:44.411028 22304 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7b37f02ae494a2dab4675da2e45ccf4: Generated new TSK 0
I20260619 14:41:44.411749 22304 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:41:44.422768 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 22216
2026-06-19T14:41:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1304 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781879837648036-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:41:44 dist-test-slave-32hj krb5kdc[22315](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:41:44 dist-test-slave-32hj krb5kdc[22315](info): set up 2 sockets
Jun 19 14:41:44 dist-test-slave-32hj krb5kdc[22315](info): commencing operation
krb5kdc: starting...
W20260619 14:41:45.372884 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781879837648036-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781879837648036-392-0/krb5kdc/test-user.keytab.
Jun 19 14:41:45 dist-test-slave-32hj krb5kdc[22315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:41:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:41:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:41:45.487409 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781879837648036-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781879837648036-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781879837648036-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39111
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39111
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:41:45.597900 22328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:41:45.598201 22328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:41:45.598243 22328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:41:45.602325 22328 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:41:45.602398 22328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:41:45.602417 22328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:41:45.602432 22328 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:41:45.602447 22328 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:41:45.607898 22328 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781879837648036-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781879837648036-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781879837648036-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:41:45.609078 22328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:41:45.610122 22328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:41:45.618927 22328 server_base.cc:1034] running on GCE node
W20260619 14:41:45.618866 22333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:45.618871 22334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:45.618886 22336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:45.619692 22328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:41:45.620857 22328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:41:45.622001 22328 hybrid_clock.cc:648] HybridClock initialized: now 1781880105621976 us; error 39 us; skew 500 ppm
Jun 19 14:41:45 dist-test-slave-32hj krb5kdc[22315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880105, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:41:45.624586 22328 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:41:45.625542 22328 webserver.cc:460] Webserver started at http://127.0.98.62:38159/ using document root <none> and password file <none>
I20260619 14:41:45.626080 22328 fs_manager.cc:359] Metadata directory not provided
I20260619 14:41:45.626144 22328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:41:45.626345 22328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:41:45.628396 22328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781879837648036-392-0/minicluster-data/master-0/data/instance:
uuid: "7e5c15a920b84fa1ab8c6423c6f85677"
format_stamp: "Formatted at 2026-06-19 14:41:45 on dist-test-slave-32hj"
server_key: "6f1499b20000ce9d737b36907ccc1d68"
server_key_iv: "9cd30bcb2ae399dccd52154c6ceb9e43"
server_key_version: "encryptionkey@0"
I20260619 14:41:45.628966 22328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781879837648036-392-0/minicluster-data/master-0/wal/instance:
uuid: "7e5c15a920b84fa1ab8c6423c6f85677"
format_stamp: "Formatted at 2026-06-19 14:41:45 on dist-test-slave-32hj"
server_key: "6f1499b20000ce9d737b36907ccc1d68"
server_key_iv: "9cd30bcb2ae399dccd52154c6ceb9e43"
server_key_version: "encryptionkey@0"
I20260619 14:41:45.632689 22328 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260619 14:41:45.635296 22343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:45.636514 22328 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:41:45.636648 22328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781879837648036-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781879837648036-392-0/minicluster-data/master-0/wal
uuid: "7e5c15a920b84fa1ab8c6423c6f85677"
format_stamp: "Formatted at 2026-06-19 14:41:45 on dist-test-slave-32hj"
server_key: "6f1499b20000ce9d737b36907ccc1d68"
server_key_iv: "9cd30bcb2ae399dccd52154c6ceb9e43"
server_key_version: "encryptionkey@0"
I20260619 14:41:45.636816 22328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781879837648036-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781879837648036-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781879837648036-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:41:45.674515 22328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:41:45.677712 22328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:41:45.677907 22328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:41:45.684840 22328 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39111
I20260619 14:41:45.684845 22395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39111 every 8 connection(s)
I20260619 14:41:45.685833 22328 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:41:45.688642 22396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:41:45.693892 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 22328
I20260619 14:41:45.694015 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781879837648036-392-0/minicluster-data/master-0/wal/instance
I20260619 14:41:45.694272 392 external_mini_cluster.cc:1428] Setting key 453eb3982a2ae4b759511cba56e63742
I20260619 14:41:45.694387 22396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e5c15a920b84fa1ab8c6423c6f85677: Bootstrap starting.
I20260619 14:41:45.696925 22396 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e5c15a920b84fa1ab8c6423c6f85677: Neither blocks nor log segments found. Creating new log.
I20260619 14:41:45.697757 22396 log.cc:824] T 00000000000000000000000000000000 P 7e5c15a920b84fa1ab8c6423c6f85677: Log is configured to *not* fsync() on all Append() calls
I20260619 14:41:45.699709 22396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e5c15a920b84fa1ab8c6423c6f85677: No bootstrap required, opened a new log
Jun 19 14:41:45 dist-test-slave-32hj krb5kdc[22315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:41:45.703603 22396 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e5c15a920b84fa1ab8c6423c6f85677 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e5c15a920b84fa1ab8c6423c6f85677" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39111 } }
I20260619 14:41:45.703863 22396 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e5c15a920b84fa1ab8c6423c6f85677 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:41:45.703910 22396 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e5c15a920b84fa1ab8c6423c6f85677 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e5c15a920b84fa1ab8c6423c6f85677, State: Initialized, Role: FOLLOWER
I20260619 14:41:45.704401 22396 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e5c15a920b84fa1ab8c6423c6f85677 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e5c15a920b84fa1ab8c6423c6f85677" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39111 } }
I20260619 14:41:45.704540 22396 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e5c15a920b84fa1ab8c6423c6f85677 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:41:45.704607 22396 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e5c15a920b84fa1ab8c6423c6f85677 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:41:45.704705 22396 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e5c15a920b84fa1ab8c6423c6f85677 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:41:45.705693 22396 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e5c15a920b84fa1ab8c6423c6f85677 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e5c15a920b84fa1ab8c6423c6f85677" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39111 } }
I20260619 14:41:45.706023 22396 leader_election.cc:302] T 00000000000000000000000000000000 P 7e5c15a920b84fa1ab8c6423c6f85677 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e5c15a920b84fa1ab8c6423c6f85677; no voters:
I20260619 14:41:45.706327 22396 leader_election.cc:288] T 00000000000000000000000000000000 P 7e5c15a920b84fa1ab8c6423c6f85677 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:41:45.706444 22401 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e5c15a920b84fa1ab8c6423c6f85677 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:41:45.706660 22401 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e5c15a920b84fa1ab8c6423c6f85677 [term 1 LEADER]: Becoming Leader. State: Replica: 7e5c15a920b84fa1ab8c6423c6f85677, State: Running, Role: LEADER
I20260619 14:41:45.707048 22401 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e5c15a920b84fa1ab8c6423c6f85677 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e5c15a920b84fa1ab8c6423c6f85677" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39111 } }
I20260619 14:41:45.707705 22396 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e5c15a920b84fa1ab8c6423c6f85677 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:41:45.709257 22399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:45.695622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48967 (local address 127.0.98.62:39111)
0619 14:41:45.696058 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:45.696073 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:45.696110 (+ 37us) server_negotiation.cc:407] Connection header received
0619 14:41:45.696789 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:45.696811 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:45.697191 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:45.697488 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:45.698394 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:45.699385 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:45.700268 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:45.700796 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:45.700950 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:45.703635 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:45.703640 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:45.703655 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:41:45.703687 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:45.705388 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:45.705815 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:45.705822 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:45.705829 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:45.705957 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:45.706219 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:45.706223 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:45.706225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:45.706837 (+ 612us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:45.707017 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:45.707716 (+ 699us) server_negotiation.cc:299] Negotiation successful
0619 14:41:45.708050 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":140,"threads_started":1}
I20260619 14:41:45.710489 22402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e5c15a920b84fa1ab8c6423c6f85677 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e5c15a920b84fa1ab8c6423c6f85677. Latest consensus state: current_term: 1 leader_uuid: "7e5c15a920b84fa1ab8c6423c6f85677" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e5c15a920b84fa1ab8c6423c6f85677" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39111 } } }
I20260619 14:41:45.711895 22402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e5c15a920b84fa1ab8c6423c6f85677 [sys.catalog]: This master's current role is: LEADER
I20260619 14:41:45.710532 22403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e5c15a920b84fa1ab8c6423c6f85677 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e5c15a920b84fa1ab8c6423c6f85677" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e5c15a920b84fa1ab8c6423c6f85677" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39111 } } }
I20260619 14:41:45.712163 22403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e5c15a920b84fa1ab8c6423c6f85677 [sys.catalog]: This master's current role is: LEADER
I20260619 14:41:45.712815 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260619 14:41:45.713444 22416 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7e5c15a920b84fa1ab8c6423c6f85677: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:41:45.713517 22416 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:41:45.713794 22418 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:41:45.715811 22418 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:41:45.721948 22418 catalog_manager.cc:1347] Generated new cluster ID: 3c6e63c3a78948d2bd684869843c6928
I20260619 14:41:45.722115 22418 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:41:45.729595 22399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:45.720326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44905 (local address 127.0.98.62:39111)
0619 14:41:45.720439 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:45.720443 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:45.720653 (+ 210us) server_negotiation.cc:407] Connection header received
0619 14:41:45.720849 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:45.720853 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:45.720897 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:45.720964 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:45.721661 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:45.722370 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:45.723570 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:45.723952 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:45.724047 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:45.725840 (+ 1793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:45.725844 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:45.725847 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:41:45.725876 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:45.727622 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:45.728122 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:45.728125 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:45.728127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:45.728224 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:45.728505 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:45.728507 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:45.728509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:45.728765 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:45.728863 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:45.729280 (+ 417us) server_negotiation.cc:299] Negotiation successful
0619 14:41:45.729420 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":32}
I20260619 14:41:45.746069 22418 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:41:45.747454 22418 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:41:45.755802 22418 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e5c15a920b84fa1ab8c6423c6f85677: Generated new TSK 0
I20260619 14:41:45.756695 22418 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:41:45.765408 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 22328
2026-06-19T14:41:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1342 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781879837648036-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:41:45 dist-test-slave-32hj krb5kdc[22427](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:41:45 dist-test-slave-32hj krb5kdc[22427](info): set up 2 sockets
Jun 19 14:41:45 dist-test-slave-32hj krb5kdc[22427](info): commencing operation
krb5kdc: starting...
W20260619 14:41:46.724687 392 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781879837648036-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781879837648036-392-0/krb5kdc/test-user.keytab.
Jun 19 14:41:46 dist-test-slave-32hj krb5kdc[22427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:41:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:41:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:41:46.839306 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781879837648036-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781879837648036-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781879837648036-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45331
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45331
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:41:46.947738 22440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:41:46.948037 22440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:41:46.948098 22440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:41:46.952268 22440 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:41:46.952348 22440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:41:46.952376 22440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:41:46.952397 22440 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:41:46.952418 22440 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:41:46.957954 22440 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781879837648036-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781879837648036-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45331
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781879837648036-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:41:46.959115 22440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:41:46.960191 22440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:41:46.969391 22446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:46.969341 22445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:46.969341 22448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:46.969480 22440 server_base.cc:1034] running on GCE node
I20260619 14:41:46.970355 22440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:41:46.971580 22440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:41:46.972770 22440 hybrid_clock.cc:648] HybridClock initialized: now 1781880106972744 us; error 40 us; skew 500 ppm
Jun 19 14:41:46 dist-test-slave-32hj krb5kdc[22427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880106, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:41:46.975535 22440 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:41:46.976457 22440 webserver.cc:460] Webserver started at http://127.0.98.62:40387/ using document root <none> and password file <none>
I20260619 14:41:46.977041 22440 fs_manager.cc:359] Metadata directory not provided
I20260619 14:41:46.977102 22440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:41:46.977281 22440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:41:46.979315 22440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781879837648036-392-0/minicluster-data/master-0/data/instance:
uuid: "148833b326b9412286467e042dc177bc"
format_stamp: "Formatted at 2026-06-19 14:41:46 on dist-test-slave-32hj"
server_key: "0c65bee2fe7705a5ded9a108652d8de6"
server_key_iv: "8f4db415897233429832d3393aec0084"
server_key_version: "encryptionkey@0"
I20260619 14:41:46.979840 22440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781879837648036-392-0/minicluster-data/master-0/wal/instance:
uuid: "148833b326b9412286467e042dc177bc"
format_stamp: "Formatted at 2026-06-19 14:41:46 on dist-test-slave-32hj"
server_key: "0c65bee2fe7705a5ded9a108652d8de6"
server_key_iv: "8f4db415897233429832d3393aec0084"
server_key_version: "encryptionkey@0"
I20260619 14:41:46.983589 22440 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260619 14:41:46.986084 22455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:46.987308 22440 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:41:46.987416 22440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781879837648036-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781879837648036-392-0/minicluster-data/master-0/wal
uuid: "148833b326b9412286467e042dc177bc"
format_stamp: "Formatted at 2026-06-19 14:41:46 on dist-test-slave-32hj"
server_key: "0c65bee2fe7705a5ded9a108652d8de6"
server_key_iv: "8f4db415897233429832d3393aec0084"
server_key_version: "encryptionkey@0"
I20260619 14:41:46.987511 22440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781879837648036-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781879837648036-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781879837648036-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:41:47.004446 22440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:41:47.007244 22440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:41:47.007416 22440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:41:47.014521 22440 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45331
I20260619 14:41:47.014583 22507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45331 every 8 connection(s)
I20260619 14:41:47.015457 22440 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:41:47.018260 22508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:41:47.023898 22508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 148833b326b9412286467e042dc177bc: Bootstrap starting.
I20260619 14:41:47.025347 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 22440
I20260619 14:41:47.025488 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781879837648036-392-0/minicluster-data/master-0/wal/instance
I20260619 14:41:47.025776 392 external_mini_cluster.cc:1428] Setting key 264f94c8d45d2f8ff4f38b224f07a7cc
I20260619 14:41:47.026722 22508 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 148833b326b9412286467e042dc177bc: Neither blocks nor log segments found. Creating new log.
I20260619 14:41:47.027514 22508 log.cc:824] T 00000000000000000000000000000000 P 148833b326b9412286467e042dc177bc: Log is configured to *not* fsync() on all Append() calls
I20260619 14:41:47.029559 22508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 148833b326b9412286467e042dc177bc: No bootstrap required, opened a new log
I20260619 14:41:47.033514 22508 raft_consensus.cc:348] T 00000000000000000000000000000000 P 148833b326b9412286467e042dc177bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "148833b326b9412286467e042dc177bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45331 } }
I20260619 14:41:47.033793 22508 raft_consensus.cc:374] T 00000000000000000000000000000000 P 148833b326b9412286467e042dc177bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:41:47.033854 22508 raft_consensus.cc:729] T 00000000000000000000000000000000 P 148833b326b9412286467e042dc177bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 148833b326b9412286467e042dc177bc, State: Initialized, Role: FOLLOWER
I20260619 14:41:47.034291 22508 consensus_queue.cc:260] T 00000000000000000000000000000000 P 148833b326b9412286467e042dc177bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "148833b326b9412286467e042dc177bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45331 } }
I20260619 14:41:47.034418 22508 raft_consensus.cc:388] T 00000000000000000000000000000000 P 148833b326b9412286467e042dc177bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:41:47.034480 22508 raft_consensus.cc:482] T 00000000000000000000000000000000 P 148833b326b9412286467e042dc177bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:41:47.034572 22508 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 148833b326b9412286467e042dc177bc [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:41:47.034579 22511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:47.026885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58835 (local address 127.0.98.62:45331)
0619 14:41:47.027299 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:47.027316 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:47.027351 (+ 35us) server_negotiation.cc:407] Connection header received
0619 14:41:47.028101 (+ 750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:47.028126 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:47.028512 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:47.028855 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:47.029176 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:47.030151 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:47.031345 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:47.031827 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:47.031941 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:47.032832 (+ 891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:47.032837 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:47.032850 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:41:47.033233 (+ 383us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:47.033287 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:41:47.033309 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:47.033380 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:47.033536 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 14:41:47.033755 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":109,"threads_started":1}
I20260619 14:41:47.035449 22508 raft_consensus.cc:504] T 00000000000000000000000000000000 P 148833b326b9412286467e042dc177bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "148833b326b9412286467e042dc177bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45331 } }
I20260619 14:41:47.035777 22508 leader_election.cc:302] T 00000000000000000000000000000000 P 148833b326b9412286467e042dc177bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 148833b326b9412286467e042dc177bc; no voters:
I20260619 14:41:47.036063 22508 leader_election.cc:288] T 00000000000000000000000000000000 P 148833b326b9412286467e042dc177bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:41:47.036139 22513 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 148833b326b9412286467e042dc177bc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:41:47.036324 22513 raft_consensus.cc:686] T 00000000000000000000000000000000 P 148833b326b9412286467e042dc177bc [term 1 LEADER]: Becoming Leader. State: Replica: 148833b326b9412286467e042dc177bc, State: Running, Role: LEADER
I20260619 14:41:47.036676 22513 consensus_queue.cc:237] T 00000000000000000000000000000000 P 148833b326b9412286467e042dc177bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "148833b326b9412286467e042dc177bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45331 } }
I20260619 14:41:47.037256 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:41:47.037374 22508 sys_catalog.cc:564] T 00000000000000000000000000000000 P 148833b326b9412286467e042dc177bc [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:41:47.040211 22515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 148833b326b9412286467e042dc177bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "148833b326b9412286467e042dc177bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "148833b326b9412286467e042dc177bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45331 } } }
I20260619 14:41:47.040470 22515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 148833b326b9412286467e042dc177bc [sys.catalog]: This master's current role is: LEADER
W20260619 14:41:47.042981 22531 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 148833b326b9412286467e042dc177bc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:41:47.043061 22531 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:41:47.043140 22534 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:41:47.043226 22514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 148833b326b9412286467e042dc177bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 148833b326b9412286467e042dc177bc. Latest consensus state: current_term: 1 leader_uuid: "148833b326b9412286467e042dc177bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "148833b326b9412286467e042dc177bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45331 } } }
I20260619 14:41:47.043336 22514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 148833b326b9412286467e042dc177bc [sys.catalog]: This master's current role is: LEADER
I20260619 14:41:47.044096 22534 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:41:47.044371 22511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:47.039962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47921 (local address 127.0.98.62:45331)
0619 14:41:47.040075 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:47.040080 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:47.040095 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:41:47.040138 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:47.040141 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:47.040187 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:47.040286 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:47.040628 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:47.041208 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:47.042295 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:47.042624 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:47.042730 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:47.043586 (+ 856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:47.043590 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:47.043593 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:41:47.043608 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:47.043638 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:41:47.043653 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:47.043725 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:47.044163 (+ 438us) server_negotiation.cc:299] Negotiation successful
0619 14:41:47.044222 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 14:41:47.050072 22534 catalog_manager.cc:1347] Generated new cluster ID: e37472dbc7f744969e195f53b2574d50
I20260619 14:41:47.050156 22534 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:41:47.070469 22534 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:41:47.071811 22534 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:41:47.081445 22534 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 148833b326b9412286467e042dc177bc: Generated new TSK 0
I20260619 14:41:47.082329 22534 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:41:47.097072 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 22440
2026-06-19T14:41:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1331 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781879837648036-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:41:47 dist-test-slave-32hj krb5kdc[22539](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:41:47 dist-test-slave-32hj krb5kdc[22539](info): set up 2 sockets
Jun 19 14:41:47 dist-test-slave-32hj krb5kdc[22539](info): commencing operation
krb5kdc: starting...
W20260619 14:41:48.052867 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781879837648036-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781879837648036-392-0/krb5kdc/test-user.keytab.
Jun 19 14:41:48 dist-test-slave-32hj krb5kdc[22539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880108, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:41:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:41:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:41:48.169828 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781879837648036-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781879837648036-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781879837648036-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35187
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35187
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:41:48.280510 22552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:41:48.280838 22552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:41:48.280884 22552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:41:48.284989 22552 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:41:48.285066 22552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:41:48.285085 22552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:41:48.285101 22552 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:41:48.285115 22552 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:41:48.290617 22552 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781879837648036-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781879837648036-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781879837648036-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:41:48.291761 22552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:41:48.292832 22552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:41:48.301339 22560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:48.301349 22557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:48.301406 22558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:48.301420 22552 server_base.cc:1034] running on GCE node
I20260619 14:41:48.302160 22552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:41:48.303247 22552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:41:48.304400 22552 hybrid_clock.cc:648] HybridClock initialized: now 1781880108304374 us; error 43 us; skew 500 ppm
Jun 19 14:41:48 dist-test-slave-32hj krb5kdc[22539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880108, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:41:48.307154 22552 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:41:48.308156 22552 webserver.cc:460] Webserver started at http://127.0.98.62:40965/ using document root <none> and password file <none>
I20260619 14:41:48.308822 22552 fs_manager.cc:359] Metadata directory not provided
I20260619 14:41:48.308900 22552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:41:48.309123 22552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:41:48.311288 22552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781879837648036-392-0/minicluster-data/master-0/data/instance:
uuid: "203c1cd9ead64c2a88adeac7e9084876"
format_stamp: "Formatted at 2026-06-19 14:41:48 on dist-test-slave-32hj"
server_key: "97e8e3d3ef0ab03f9d2b23bd3ba42bd7"
server_key_iv: "0c416399dd5c5845f2afa750f5322102"
server_key_version: "encryptionkey@0"
I20260619 14:41:48.311868 22552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781879837648036-392-0/minicluster-data/master-0/wal/instance:
uuid: "203c1cd9ead64c2a88adeac7e9084876"
format_stamp: "Formatted at 2026-06-19 14:41:48 on dist-test-slave-32hj"
server_key: "97e8e3d3ef0ab03f9d2b23bd3ba42bd7"
server_key_iv: "0c416399dd5c5845f2afa750f5322102"
server_key_version: "encryptionkey@0"
I20260619 14:41:48.315807 22552 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 14:41:48.318368 22567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:48.319633 22552 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:41:48.319749 22552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781879837648036-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781879837648036-392-0/minicluster-data/master-0/wal
uuid: "203c1cd9ead64c2a88adeac7e9084876"
format_stamp: "Formatted at 2026-06-19 14:41:48 on dist-test-slave-32hj"
server_key: "97e8e3d3ef0ab03f9d2b23bd3ba42bd7"
server_key_iv: "0c416399dd5c5845f2afa750f5322102"
server_key_version: "encryptionkey@0"
I20260619 14:41:48.319850 22552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781879837648036-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781879837648036-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781879837648036-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:41:48.324466 22552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:41:48.327745 22552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:41:48.327920 22552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:41:48.334702 22552 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35187
I20260619 14:41:48.334703 22619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35187 every 8 connection(s)
I20260619 14:41:48.335693 22552 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:41:48.338433 22620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:41:48.343735 22620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 203c1cd9ead64c2a88adeac7e9084876: Bootstrap starting.
I20260619 14:41:48.345888 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 22552
I20260619 14:41:48.346026 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781879837648036-392-0/minicluster-data/master-0/wal/instance
I20260619 14:41:48.346284 22620 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 203c1cd9ead64c2a88adeac7e9084876: Neither blocks nor log segments found. Creating new log.
I20260619 14:41:48.346326 392 external_mini_cluster.cc:1428] Setting key bdc2c9f9c5209a15b7010997118e01fd
I20260619 14:41:48.347103 22620 log.cc:824] T 00000000000000000000000000000000 P 203c1cd9ead64c2a88adeac7e9084876: Log is configured to *not* fsync() on all Append() calls
I20260619 14:41:48.349442 22620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 203c1cd9ead64c2a88adeac7e9084876: No bootstrap required, opened a new log
I20260619 14:41:48.351552 22624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:48.347426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35909 (local address 127.0.98.62:35187)
0619 14:41:48.347822 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:48.347836 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:48.347871 (+ 35us) server_negotiation.cc:407] Connection header received
0619 14:41:48.348757 (+ 886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:48.348782 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:48.349133 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:48.349482 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:48.350177 (+ 695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:48.350183 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:48.350196 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:41:48.350207 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:48.350269 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:41:48.350296 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:48.350399 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:48.350466 (+ 67us) server_negotiation.cc:299] Negotiation successful
0619 14:41:48.350622 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":118,"threads_started":1}
I20260619 14:41:48.353775 22620 raft_consensus.cc:348] T 00000000000000000000000000000000 P 203c1cd9ead64c2a88adeac7e9084876 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "203c1cd9ead64c2a88adeac7e9084876" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35187 } }
I20260619 14:41:48.354054 22620 raft_consensus.cc:374] T 00000000000000000000000000000000 P 203c1cd9ead64c2a88adeac7e9084876 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:41:48.354111 22620 raft_consensus.cc:729] T 00000000000000000000000000000000 P 203c1cd9ead64c2a88adeac7e9084876 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 203c1cd9ead64c2a88adeac7e9084876, State: Initialized, Role: FOLLOWER
I20260619 14:41:48.354447 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:41:48.354549 22620 consensus_queue.cc:260] T 00000000000000000000000000000000 P 203c1cd9ead64c2a88adeac7e9084876 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "203c1cd9ead64c2a88adeac7e9084876" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35187 } }
I20260619 14:41:48.354691 22620 raft_consensus.cc:388] T 00000000000000000000000000000000 P 203c1cd9ead64c2a88adeac7e9084876 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:41:48.354748 22620 raft_consensus.cc:482] T 00000000000000000000000000000000 P 203c1cd9ead64c2a88adeac7e9084876 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:41:48.354839 22620 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 203c1cd9ead64c2a88adeac7e9084876 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:41:48.355808 22620 raft_consensus.cc:504] T 00000000000000000000000000000000 P 203c1cd9ead64c2a88adeac7e9084876 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "203c1cd9ead64c2a88adeac7e9084876" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35187 } }
I20260619 14:41:48.356146 22620 leader_election.cc:302] T 00000000000000000000000000000000 P 203c1cd9ead64c2a88adeac7e9084876 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 203c1cd9ead64c2a88adeac7e9084876; no voters:
I20260619 14:41:48.356482 22620 leader_election.cc:288] T 00000000000000000000000000000000 P 203c1cd9ead64c2a88adeac7e9084876 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:41:48.356540 22629 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 203c1cd9ead64c2a88adeac7e9084876 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:41:48.356840 22629 raft_consensus.cc:686] T 00000000000000000000000000000000 P 203c1cd9ead64c2a88adeac7e9084876 [term 1 LEADER]: Becoming Leader. State: Replica: 203c1cd9ead64c2a88adeac7e9084876, State: Running, Role: LEADER
I20260619 14:41:48.357187 22629 consensus_queue.cc:237] T 00000000000000000000000000000000 P 203c1cd9ead64c2a88adeac7e9084876 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "203c1cd9ead64c2a88adeac7e9084876" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35187 } }
I20260619 14:41:48.357934 22620 sys_catalog.cc:564] T 00000000000000000000000000000000 P 203c1cd9ead64c2a88adeac7e9084876 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:41:48.358261 22624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:48.356448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51915 (local address 127.0.98.62:35187)
0619 14:41:48.356559 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:48.356563 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:48.356852 (+ 289us) server_negotiation.cc:407] Connection header received
0619 14:41:48.356998 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:48.357001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:48.357042 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:48.357129 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:48.357837 (+ 708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:48.357841 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:48.357844 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:41:48.357855 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:48.357885 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:41:48.357897 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:48.357979 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:48.358104 (+ 125us) server_negotiation.cc:299] Negotiation successful
0619 14:41:48.358145 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 14:41:48.358435 22629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 203c1cd9ead64c2a88adeac7e9084876 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 203c1cd9ead64c2a88adeac7e9084876. Latest consensus state: current_term: 1 leader_uuid: "203c1cd9ead64c2a88adeac7e9084876" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "203c1cd9ead64c2a88adeac7e9084876" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35187 } } }
I20260619 14:41:48.358533 22629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 203c1cd9ead64c2a88adeac7e9084876 [sys.catalog]: This master's current role is: LEADER
I20260619 14:41:48.358716 22629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 203c1cd9ead64c2a88adeac7e9084876 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "203c1cd9ead64c2a88adeac7e9084876" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "203c1cd9ead64c2a88adeac7e9084876" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35187 } } }
I20260619 14:41:48.358783 22629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 203c1cd9ead64c2a88adeac7e9084876 [sys.catalog]: This master's current role is: LEADER
W20260619 14:41:48.362703 22645 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 203c1cd9ead64c2a88adeac7e9084876: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:41:48.362769 22645 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:41:48.362851 22634 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:41:48.363514 22634 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:41:48.368875 22634 catalog_manager.cc:1347] Generated new cluster ID: b71a5dbaea0c4de19375d1a96cc267fb
I20260619 14:41:48.368947 22634 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:41:48.380573 22634 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:41:48.380929 22634 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:41:48.395100 22634 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 203c1cd9ead64c2a88adeac7e9084876: Generated new TSK 0
I20260619 14:41:48.395974 22634 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:41:48.405248 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 22552
2026-06-19T14:41:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1308 ms)
[----------] 18 tests from AuthTokenIssuingTest (23827 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:41:48 dist-test-slave-32hj krb5kdc[22651](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:41:48 dist-test-slave-32hj krb5kdc[22651](info): set up 2 sockets
Jun 19 14:41:48 dist-test-slave-32hj krb5kdc[22651](info): commencing operation
krb5kdc: starting...
W20260619 14:41:49.360798 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/krb5kdc/test-user.keytab.
Jun 19 14:41:49 dist-test-slave-32hj krb5kdc[22651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:41:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:41:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:41:49.479156 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38917
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38917,127.0.98.61:45343,127.0.98.60:42439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:41:49.587913 22664 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 14:41:49.588201 22664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:41:49.588227 22664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:41:49.588284 22664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:41:49.592216 22664 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:41:49.592281 22664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:41:49.592300 22664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:41:49.592315 22664 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:41:49.592342 22664 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:41:49.597640 22664 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38917,127.0.98.61:45343,127.0.98.60:42439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:41:49.598723 22664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:41:49.599745 22664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:41:49.608544 22669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:49.608628 22672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:49.608552 22670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:49.608764 22664 server_base.cc:1034] running on GCE node
I20260619 14:41:49.609421 22664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:41:49.610523 22664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:41:49.611709 22664 hybrid_clock.cc:648] HybridClock initialized: now 1781880109611679 us; error 42 us; skew 500 ppm
Jun 19 14:41:49 dist-test-slave-32hj krb5kdc[22651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880109, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:41:49.619400 22664 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:41:49.620362 22664 webserver.cc:460] Webserver started at http://127.0.98.62:41243/ using document root <none> and password file <none>
I20260619 14:41:49.620960 22664 fs_manager.cc:359] Metadata directory not provided
I20260619 14:41:49.621030 22664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:41:49.621244 22664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:41:49.623353 22664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/minicluster-data/master-0/data/instance:
uuid: "43ed6e24011642cd988958cde97c58da"
format_stamp: "Formatted at 2026-06-19 14:41:49 on dist-test-slave-32hj"
server_key: "9920ef0b770434150863eac81bcc6387"
server_key_iv: "7187e8834302488f4d723c71d2f3a0f1"
server_key_version: "encryptionkey@0"
I20260619 14:41:49.623888 22664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/minicluster-data/master-0/wal/instance:
uuid: "43ed6e24011642cd988958cde97c58da"
format_stamp: "Formatted at 2026-06-19 14:41:49 on dist-test-slave-32hj"
server_key: "9920ef0b770434150863eac81bcc6387"
server_key_iv: "7187e8834302488f4d723c71d2f3a0f1"
server_key_version: "encryptionkey@0"
I20260619 14:41:49.627596 22664 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260619 14:41:49.629997 22679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:49.631290 22664 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:41:49.631428 22664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/minicluster-data/master-0/wal
uuid: "43ed6e24011642cd988958cde97c58da"
format_stamp: "Formatted at 2026-06-19 14:41:49 on dist-test-slave-32hj"
server_key: "9920ef0b770434150863eac81bcc6387"
server_key_iv: "7187e8834302488f4d723c71d2f3a0f1"
server_key_version: "encryptionkey@0"
I20260619 14:41:49.631536 22664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:41:49.645437 22664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:41:49.648343 22664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:41:49.648522 22664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:41:49.654942 22664 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38917
I20260619 14:41:49.655259 22731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38917 every 8 connection(s)
I20260619 14:41:49.655869 22664 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:41:49.658524 22732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:41:49.662089 22732 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38917 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:41:49.665604 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 22664
I20260619 14:41:49.665732 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/minicluster-data/master-0/wal/instance
I20260619 14:41:49.666013 392 external_mini_cluster.cc:1428] Setting key b30ac5215d2e1e3f2249c0e231e649ad
Jun 19 14:41:49 dist-test-slave-32hj krb5kdc[22651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880109, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260619 14:41:49.677544 22735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:49.664230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46884 (local address 127.0.98.62:38917)
0619 14:41:49.664686 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:49.664705 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:49.664887 (+ 182us) server_negotiation.cc:407] Connection header received
0619 14:41:49.666197 (+ 1310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:49.666202 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:49.666382 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:49.666535 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:49.667589 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:49.668894 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:49.669886 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:49.670444 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:49.670620 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:49.672863 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:49.672870 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:49.672886 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:41:49.672915 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:49.674524 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:49.674937 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:49.674944 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:49.674945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:49.675048 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:49.675319 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:49.675323 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:49.675324 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:49.675711 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:49.675827 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:49.676205 (+ 378us) server_negotiation.cc:299] Negotiation successful
0619 14:41:49.676470 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":314,"spinlock_wait_cycles":49920,"thread_start_us":164,"threads_started":1}
I20260619 14:41:49.677544 22734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:49.664230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38917 (local address 127.0.0.1:46884)
0619 14:41:49.664749 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:49.664791 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:49.665825 (+ 1034us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:49.666585 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:49.666590 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:49.667014 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:41:49.667395 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:49.667415 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:49.669002 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:49.669005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:49.669768 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:49.669778 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:49.670614 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:49.670618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:49.670715 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:49.671297 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:41:49.671313 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:41:49.672686 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:41:49.674616 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:41:49.674624 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:41:49.674635 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:41:49.674841 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:41:49.675132 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:41:49.675135 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:41:49.675137 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:41:49.675240 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:41:49.675815 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:41:49.675822 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:41:49.675956 (+ 134us) client_negotiation.cc:769] Sending connection context
0619 14:41:49.676135 (+ 179us) client_negotiation.cc:240] Negotiation successful
0619 14:41:49.676527 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":93,"thread_start_us":119,"threads_started":1}
I20260619 14:41:49.680123 22732 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45343 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:41:49.680944 22734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:49.680661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45343 (local address 127.0.0.1:37530)
0619 14:41:49.680813 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:49.680881 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260619 14:41:49.681144 22682 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45343: connect: Connection refused (error 111)
I20260619 14:41:49.681995 22734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:49.681757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45343 (local address 127.0.0.1:37538)
0619 14:41:49.681882 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:49.681929 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260619 14:41:49.682130 22732 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:45343: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45343: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260619 14:41:49.689162 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:45343
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38917,127.0.98.61:45343,127.0.98.60:42439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:41:49.731403 22732 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45343 } attempt: 1
I20260619 14:41:49.732309 22734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:49.732008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45343 (local address 127.0.0.1:37552)
0619 14:41:49.732161 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:49.732229 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 14:41:49.733028 22734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:49.732822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45343 (local address 127.0.0.1:37556)
0619 14:41:49.732917 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:49.732962 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260619 14:41:49.733144 22732 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:45343: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45343: connect: Connection refused (error 111)
W20260619 14:41:49.798615 22739 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 14:41:49.798938 22739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:41:49.798974 22739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:41:49.799038 22739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 14:41:49.801332 22732 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45343 } attempt: 2
I20260619 14:41:49.802249 22734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:49.801910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45343 (local address 127.0.0.1:37568)
0619 14:41:49.802105 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:49.802172 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260619 14:41:49.802927 22734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:49.802725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45343 (local address 127.0.0.1:37576)
0619 14:41:49.802825 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:49.802869 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260619 14:41:49.803036 22732 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:45343: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45343: connect: Connection refused (error 111)
W20260619 14:41:49.803375 22739 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:41:49.803431 22739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:41:49.803460 22739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:41:49.803488 22739 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:41:49.803519 22739 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:41:49.808931 22739 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38917,127.0.98.61:45343,127.0.98.60:42439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:45343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:41:49.809944 22739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:41:49.810910 22739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:41:49.818866 22744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:49.818950 22745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:49.818964 22739 server_base.cc:1034] running on GCE node
W20260619 14:41:49.818866 22747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:49.819716 22739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:41:49.820801 22739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:41:49.821952 22739 hybrid_clock.cc:648] HybridClock initialized: now 1781880109821935 us; error 28 us; skew 500 ppm
Jun 19 14:41:49 dist-test-slave-32hj krb5kdc[22651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880109, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:41:49.824556 22739 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260619 14:41:49.825409 22739 webserver.cc:460] Webserver started at http://127.0.98.61:40225/ using document root <none> and password file <none>
I20260619 14:41:49.825912 22739 fs_manager.cc:359] Metadata directory not provided
I20260619 14:41:49.825973 22739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:41:49.826170 22739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:41:49.828157 22739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/minicluster-data/master-1/data/instance:
uuid: "26ab6d6de7bf404c9f92af9f38d3252a"
format_stamp: "Formatted at 2026-06-19 14:41:49 on dist-test-slave-32hj"
server_key: "65fd24612f9932a5f443a77941ced34a"
server_key_iv: "bf8a1ba33b36f39268bc39ae55e142ff"
server_key_version: "encryptionkey@0"
I20260619 14:41:49.828693 22739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/minicluster-data/master-1/wal/instance:
uuid: "26ab6d6de7bf404c9f92af9f38d3252a"
format_stamp: "Formatted at 2026-06-19 14:41:49 on dist-test-slave-32hj"
server_key: "65fd24612f9932a5f443a77941ced34a"
server_key_iv: "bf8a1ba33b36f39268bc39ae55e142ff"
server_key_version: "encryptionkey@0"
I20260619 14:41:49.832170 22739 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:41:49.834456 22754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:49.835651 22739 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 14:41:49.835767 22739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/minicluster-data/master-1/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/minicluster-data/master-1/wal
uuid: "26ab6d6de7bf404c9f92af9f38d3252a"
format_stamp: "Formatted at 2026-06-19 14:41:49 on dist-test-slave-32hj"
server_key: "65fd24612f9932a5f443a77941ced34a"
server_key_iv: "bf8a1ba33b36f39268bc39ae55e142ff"
server_key_version: "encryptionkey@0"
I20260619 14:41:49.835855 22739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:41:49.848397 22739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:41:49.850975 22739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:41:49.851151 22739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:41:49.857436 22739 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:45343
I20260619 14:41:49.857663 22806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:45343 every 8 connection(s)
I20260619 14:41:49.858345 22739 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/minicluster-data/master-1/data/info.pb
I20260619 14:41:49.860999 22807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:41:49.864769 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 22739
I20260619 14:41:49.864555 22807 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38917 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:41:49.864872 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/minicluster-data/master-1/wal/instance
I20260619 14:41:49.865123 392 external_mini_cluster.cc:1428] Setting key 4fd70e4b05b3188fde698d536be4f960
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 19 14:41:49 dist-test-slave-32hj krb5kdc[22651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880109, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:41:49.878660 22735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:49.866471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46896 (local address 127.0.98.62:38917)
0619 14:41:49.866610 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:49.866615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:49.867501 (+ 886us) server_negotiation.cc:407] Connection header received
0619 14:41:49.868417 (+ 916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:49.868421 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:49.868480 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:49.868559 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:49.869682 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:49.870198 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:49.871262 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:49.871492 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:49.871586 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:49.875197 (+ 3611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:49.875200 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:49.875202 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:41:49.875228 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:49.876687 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:49.877177 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:49.877181 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:49.877183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:49.877259 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:49.877559 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:49.877562 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:49.877564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:49.877747 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:49.877831 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:49.878355 (+ 524us) server_negotiation.cc:299] Negotiation successful
0619 14:41:49.878481 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 14:41:49.879443 22810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:49.866835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38917 (local address 127.0.0.1:46896)
0619 14:41:49.867333 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:49.867388 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:49.868218 (+ 830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:49.868747 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:49.868756 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:49.869091 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:41:49.869544 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:49.869557 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:49.870308 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:49.870311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:49.871149 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:49.871158 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:49.871630 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:49.871634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:49.871799 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:49.872344 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:41:49.872360 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:41:49.875031 (+ 2671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:41:49.876806 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:41:49.876813 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:41:49.876829 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:41:49.877086 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:41:49.877338 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:41:49.877341 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:41:49.877343 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:41:49.877471 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:41:49.877843 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:41:49.877850 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:41:49.878115 (+ 265us) client_negotiation.cc:769] Sending connection context
0619 14:41:49.878280 (+ 165us) client_negotiation.cc:240] Negotiation successful
0619 14:41:49.878610 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":111,"threads_started":1}
I20260619 14:41:49.881053 22807 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45343 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:41:49 dist-test-slave-32hj krb5kdc[22651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880109, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260619 14:41:49.889421 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:42439
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38917,127.0.98.61:45343,127.0.98.60:42439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:41:49.889813 22810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:49.881544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45343 (local address 127.0.0.1:37592)
0619 14:41:49.881649 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:49.881665 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:49.881731 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:49.882305 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:49.882307 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:49.882448 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:41:49.882537 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:49.882542 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:49.883590 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:49.883593 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:49.884145 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:49.884154 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:49.884844 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:49.884848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:49.884960 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:49.885383 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:41:49.885393 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:41:49.886533 (+ 1140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:41:49.888283 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:41:49.888286 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:41:49.888289 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:41:49.888535 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:41:49.888814 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:41:49.888817 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:41:49.888818 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:41:49.888891 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:41:49.889389 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:41:49.889391 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:41:49.889467 (+ 76us) client_negotiation.cc:769] Sending connection context
0619 14:41:49.889546 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 14:41:49.889688 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":19}
I20260619 14:41:49.890295 22812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:49.881734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37592 (local address 127.0.98.61:45343)
0619 14:41:49.881977 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:49.881986 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:49.882003 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:41:49.882032 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:49.882036 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:49.882187 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:49.882320 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:49.882626 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:49.883479 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:49.884252 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:49.884741 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:49.884847 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:49.886669 (+ 1822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:49.886678 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:49.886689 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:41:49.886709 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:49.888191 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:49.888632 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:49.888636 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:49.888637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:49.888712 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:49.888966 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:49.888969 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:49.888971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:49.889291 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:49.889431 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:49.890038 (+ 607us) server_negotiation.cc:299] Negotiation successful
0619 14:41:49.890177 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":174,"thread_start_us":86,"threads_started":1}
I20260619 14:41:49.892331 22807 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42439 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:41:49.892987 22810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:49.892761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42439 (local address 127.0.0.1:55646)
0619 14:41:49.892866 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:49.892934 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 14:41:49.893162 22757 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42439: connect: Connection refused (error 111)
I20260619 14:41:49.893947 22810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:49.893762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42439 (local address 127.0.0.1:55658)
0619 14:41:49.893856 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:49.893899 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260619 14:41:49.894058 22807 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42439: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42439: connect: Connection refused (error 111)
I20260619 14:41:49.894192 22732 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45343 } attempt: 3
Jun 19 14:41:49 dist-test-slave-32hj krb5kdc[22651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880109, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 14:41:49.902406 22812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:49.894624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37602 (local address 127.0.98.61:45343)
0619 14:41:49.894724 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:49.894728 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:49.894744 (+ 16us) server_negotiation.cc:407] Connection header received
0619 14:41:49.894879 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:49.894881 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:49.894914 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:49.894971 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:49.895428 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:49.895912 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:49.896991 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:49.897241 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:49.897328 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:49.899290 (+ 1962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:49.899293 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:49.899295 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:41:49.899315 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:49.900875 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:49.901330 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:49.901333 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:49.901334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:49.901408 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:49.901686 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:49.901689 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:49.901690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:49.901872 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:49.901959 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:49.902200 (+ 241us) server_negotiation.cc:299] Negotiation successful
0619 14:41:49.902293 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 14:41:49.902454 22734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:49.894556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45343 (local address 127.0.0.1:37602)
0619 14:41:49.894690 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:49.894706 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:49.894796 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:49.895011 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:49.895013 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:49.895203 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:41:49.895332 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:49.895340 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:49.896023 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:49.896026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:49.896878 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:49.896889 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:49.897340 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:49.897344 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:49.897446 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:49.897887 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:41:49.897900 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:41:49.899143 (+ 1243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:41:49.900974 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:41:49.900977 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:41:49.900980 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:41:49.901236 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:41:49.901481 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:41:49.901484 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:41:49.901486 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:41:49.901573 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:41:49.901957 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:41:49.901960 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:41:49.902061 (+ 101us) client_negotiation.cc:769] Sending connection context
0619 14:41:49.902160 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 14:41:49.902292 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260619 14:41:49.903214 22732 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42439 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:41:49.903859 22734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:49.903638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42439 (local address 127.0.0.1:55660)
0619 14:41:49.903750 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:49.903799 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 14:41:49.904482 22734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:49.904221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42439 (local address 127.0.0.1:55670)
0619 14:41:49.904385 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:49.904431 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260619 14:41:49.904606 22732 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42439: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42439: connect: Connection refused (error 111)
I20260619 14:41:49.935750 22732 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42439 } attempt: 1
I20260619 14:41:49.936537 22734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:49.936254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42439 (local address 127.0.0.1:55676)
0619 14:41:49.936394 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:49.936458 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 14:41:49.937219 22734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:49.937035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42439 (local address 127.0.0.1:55678)
0619 14:41:49.937117 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:49.937161 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260619 14:41:49.937333 22732 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42439: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42439: connect: Connection refused (error 111)
I20260619 14:41:49.943306 22807 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42439 } attempt: 1
I20260619 14:41:49.944217 22810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:49.943887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42439 (local address 127.0.0.1:55680)
0619 14:41:49.944073 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:49.944138 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260619 14:41:49.945415 22810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:49.945194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42439 (local address 127.0.0.1:55686)
0619 14:41:49.945306 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:49.945356 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 14:41:49.945534 22807 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42439: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42439: connect: Connection refused (error 111)
W20260619 14:41:50.002851 22813 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 14:41:50.003151 22813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:41:50.003185 22813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:41:50.003247 22813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:41:50.007207 22813 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:41:50.007270 22813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:41:50.007296 22813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:41:50.007316 22813 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:41:50.007336 22813 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:41:50.012519 22732 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42439 } attempt: 2
I20260619 14:41:50.012554 22813 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38917,127.0.98.61:45343,127.0.98.60:42439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:42439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:41:50.013336 22734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:50.013054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42439 (local address 127.0.0.1:55702)
0619 14:41:50.013198 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:50.013260 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 14:41:50.013706 22813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:41:50.013684 22807 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42439 } attempt: 2
I20260619 14:41:50.014384 22810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:50.014118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42439 (local address 127.0.0.1:55704)
0619 14:41:50.014252 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:50.014312 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 14:41:50.014469 22734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:50.014205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42439 (local address 127.0.0.1:55716)
0619 14:41:50.014339 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:50.014397 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 14:41:50.014596 22732 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42439: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42439: connect: Connection refused (error 111)
I20260619 14:41:50.015000 22813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:41:50.015250 22810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:50.014969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42439 (local address 127.0.0.1:55722)
0619 14:41:50.015124 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:50.015192 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260619 14:41:50.015647 22807 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42439: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42439: connect: Connection refused (error 111)
W20260619 14:41:50.023278 22819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:50.023269 22822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:50.023269 22820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:50.023568 22813 server_base.cc:1034] running on GCE node
I20260619 14:41:50.024108 22813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:41:50.025210 22813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:41:50.026364 22813 hybrid_clock.cc:648] HybridClock initialized: now 1781880110026341 us; error 36 us; skew 500 ppm
Jun 19 14:41:50 dist-test-slave-32hj krb5kdc[22651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880110, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:41:50.028878 22813 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260619 14:41:50.029767 22813 webserver.cc:460] Webserver started at http://127.0.98.60:43015/ using document root <none> and password file <none>
I20260619 14:41:50.030287 22813 fs_manager.cc:359] Metadata directory not provided
I20260619 14:41:50.030349 22813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:41:50.030544 22813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:41:50.032502 22813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/minicluster-data/master-2/data/instance:
uuid: "f78f6d1d2a964a0d9828228d39d29619"
format_stamp: "Formatted at 2026-06-19 14:41:50 on dist-test-slave-32hj"
server_key: "f9210f699de9213443dcebf94c441992"
server_key_iv: "65aefbabde8034dca1216f201f7ff5aa"
server_key_version: "encryptionkey@0"
I20260619 14:41:50.033066 22813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/minicluster-data/master-2/wal/instance:
uuid: "f78f6d1d2a964a0d9828228d39d29619"
format_stamp: "Formatted at 2026-06-19 14:41:50 on dist-test-slave-32hj"
server_key: "f9210f699de9213443dcebf94c441992"
server_key_iv: "65aefbabde8034dca1216f201f7ff5aa"
server_key_version: "encryptionkey@0"
I20260619 14:41:50.036696 22813 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:41:50.039126 22829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:50.040380 22813 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:41:50.040486 22813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/minicluster-data/master-2/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/minicluster-data/master-2/wal
uuid: "f78f6d1d2a964a0d9828228d39d29619"
format_stamp: "Formatted at 2026-06-19 14:41:50 on dist-test-slave-32hj"
server_key: "f9210f699de9213443dcebf94c441992"
server_key_iv: "65aefbabde8034dca1216f201f7ff5aa"
server_key_version: "encryptionkey@0"
I20260619 14:41:50.040596 22813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:41:50.055445 22813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:41:50.058776 22813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:41:50.058960 22813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:41:50.065549 22813 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:42439
I20260619 14:41:50.065804 22881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:42439 every 8 connection(s)
I20260619 14:41:50.066509 22813 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/minicluster-data/master-2/data/info.pb
I20260619 14:41:50.069226 22882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:41:50.072888 22882 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38917 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:41:50.075765 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 22813
I20260619 14:41:50.075877 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781879837648036-392-0/minicluster-data/master-2/wal/instance
I20260619 14:41:50.076153 392 external_mini_cluster.cc:1428] Setting key d30b2543b7c30b1e69f6c1d3666e33b8
Jun 19 14:41:50 dist-test-slave-32hj krb5kdc[22651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880110, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 19 14:41:50 dist-test-slave-32hj krb5kdc[22651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 14:41:50.086439 22735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:50.074653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46898 (local address 127.0.98.62:38917)
0619 14:41:50.074800 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:50.074803 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:50.075614 (+ 811us) server_negotiation.cc:407] Connection header received
0619 14:41:50.076760 (+ 1146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:50.076764 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:50.076823 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:50.076894 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:50.078287 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:50.078784 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:50.079733 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:50.079941 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:50.080026 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:50.082652 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:50.082662 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:50.082665 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:41:50.082692 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:50.084285 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:50.084813 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:50.084817 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:50.084819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:50.084911 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:50.085234 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:50.085237 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:50.085238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:50.085452 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:50.085550 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:50.086143 (+ 593us) server_negotiation.cc:299] Negotiation successful
0619 14:41:50.086272 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 14:41:50.087323 22886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:50.076814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55736 (local address 127.0.98.60:42439)
0619 14:41:50.077146 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:50.077155 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:50.077281 (+ 126us) server_negotiation.cc:407] Connection header received
0619 14:41:50.077373 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:50.077378 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:50.077572 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:50.077737 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:50.078366 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:50.079238 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:50.080032 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:50.080422 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:50.080521 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:50.083444 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:50.083452 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:50.083466 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:41:50.083491 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:50.085067 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:50.085468 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:50.085474 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:50.085476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:50.085557 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:50.085811 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:50.085813 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:50.085814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:50.086028 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:50.086246 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:50.086437 (+ 191us) server_negotiation.cc:299] Negotiation successful
0619 14:41:50.086582 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":243,"spinlock_wait_cycles":2432,"thread_start_us":119,"threads_started":1}
I20260619 14:41:50.087321 22884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:50.074990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38917 (local address 127.0.0.1:46898)
0619 14:41:50.075433 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:50.075475 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:50.076513 (+ 1038us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:50.077088 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:50.077097 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:50.077551 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:41:50.078150 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:50.078163 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:50.078880 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:50.078884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:50.079621 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:50.079629 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:50.080067 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:50.080071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:50.080290 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:50.080926 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:41:50.080948 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:41:50.082501 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:41:50.084405 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:41:50.084412 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:41:50.084420 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:41:50.084699 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:41:50.084995 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:41:50.084997 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:41:50.084998 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:41:50.085138 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:41:50.085554 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:41:50.085559 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:41:50.085907 (+ 348us) client_negotiation.cc:769] Sending connection context
0619 14:41:50.086066 (+ 159us) client_negotiation.cc:240] Negotiation successful
0619 14:41:50.086391 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":63,"thread_start_us":113,"threads_started":1}
I20260619 14:41:50.088548 22882 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45343 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:41:50 dist-test-slave-32hj krb5kdc[22651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880110, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 19 14:41:50 dist-test-slave-32hj krb5kdc[22651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:41:50.096405 22812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:50.089119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37610 (local address 127.0.98.61:45343)
0619 14:41:50.089241 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:50.089244 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:50.089257 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:41:50.089355 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:50.089358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:50.089405 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:50.089487 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:50.089939 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:50.090444 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:50.091165 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:50.091376 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:50.091464 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:50.093533 (+ 2069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:50.093536 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:50.093538 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:41:50.093562 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:50.094918 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:50.095330 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:50.095334 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:50.095336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:50.095416 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:50.095649 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:50.095653 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:50.095655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:50.095838 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:50.095922 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:50.096173 (+ 251us) server_negotiation.cc:299] Negotiation successful
0619 14:41:50.096269 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 14:41:50.096416 22884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:50.089024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45343 (local address 127.0.0.1:37610)
0619 14:41:50.089186 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:50.089200 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:50.089271 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:50.089494 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:50.089496 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:50.089696 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:41:50.089823 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:50.089831 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:50.090539 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:50.090543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:50.091054 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:50.091061 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:50.091469 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:50.091472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:50.091565 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:50.092005 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:41:50.092017 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:41:50.093400 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:41:50.095027 (+ 1627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:41:50.095030 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:41:50.095032 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:41:50.095237 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:41:50.095494 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:41:50.095497 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:41:50.095500 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:41:50.095574 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:41:50.095922 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:41:50.095925 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:41:50.096014 (+ 89us) client_negotiation.cc:769] Sending connection context
0619 14:41:50.096122 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 14:41:50.096255 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
I20260619 14:41:50.097136 22882 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42439 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:41:50.097726 22735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:50.089785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46906 (local address 127.0.98.62:38917)
0619 14:41:50.089892 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:50.089896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:50.089908 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:41:50.089999 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:50.090002 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:50.090040 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:50.090102 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:50.090671 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:50.091144 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:50.091874 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:50.092089 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:50.092179 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:50.094651 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:50.094655 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:50.094658 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:41:50.094683 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:50.095975 (+ 1292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:50.096408 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:50.096412 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:50.096413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:50.096636 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:50.096914 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:50.096917 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:50.096919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:50.097122 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:50.097348 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:50.097478 (+ 130us) server_negotiation.cc:299] Negotiation successful
0619 14:41:50.097578 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Jun 19 14:41:50 dist-test-slave-32hj krb5kdc[22651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880110, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 19 14:41:50 dist-test-slave-32hj krb5kdc[22651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 14:41:50.104485 22884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:50.097647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42439 (local address 127.0.0.1:55740)
0619 14:41:50.097747 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:50.097760 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:50.097826 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:50.098084 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:50.098086 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:50.098227 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:41:50.098289 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:50.098293 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:50.099046 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:50.099049 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:50.099549 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:50.099555 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:50.099957 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:50.099960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:50.100026 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:50.100424 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:41:50.100434 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:41:50.101584 (+ 1150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:41:50.103085 (+ 1501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:41:50.103088 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:41:50.103090 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:41:50.103319 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:41:50.103570 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:41:50.103572 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:41:50.103574 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:41:50.103646 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:41:50.104020 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:41:50.104022 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:41:50.104099 (+ 77us) client_negotiation.cc:769] Sending connection context
0619 14:41:50.104184 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 14:41:50.104322 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260619 14:41:50.104712 22886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:50.097846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55740 (local address 127.0.98.60:42439)
0619 14:41:50.097935 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:50.097938 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:50.097948 (+ 10us) server_negotiation.cc:407] Connection header received
0619 14:41:50.097971 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:50.097973 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:50.098002 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:50.098060 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:50.098380 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:50.098959 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:50.099642 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:50.099872 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:50.099953 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:50.101701 (+ 1748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:50.101704 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:50.101707 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:41:50.101728 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:50.102983 (+ 1255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:50.103406 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:50.103409 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:50.103411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:50.103491 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:50.103737 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:50.103739 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:50.103740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:50.103929 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:50.104025 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:50.104246 (+ 221us) server_negotiation.cc:299] Negotiation successful
0619 14:41:50.104388 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":36}
I20260619 14:41:50.106491 22812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:50.098752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37626 (local address 127.0.98.61:45343)
0619 14:41:50.098857 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:50.098860 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:50.098873 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:41:50.098938 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:50.098941 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:50.098977 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:50.099039 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:50.099574 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:50.100041 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:50.100890 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:50.101097 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:50.101184 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:50.103471 (+ 2287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:50.103475 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:50.103477 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:41:50.103500 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:50.105059 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:50.105422 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:50.105425 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:50.105427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:50.105501 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:50.105736 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:50.105739 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:50.105741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:50.105902 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:50.106114 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:50.106228 (+ 114us) server_negotiation.cc:299] Negotiation successful
0619 14:41:50.106330 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 14:41:50.106807 22807 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42439 } attempt: 3
I20260619 14:41:50.108667 22882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f78f6d1d2a964a0d9828228d39d29619: Bootstrap starting.
I20260619 14:41:50.109797 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:41:50.111410 22882 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f78f6d1d2a964a0d9828228d39d29619: Neither blocks nor log segments found. Creating new log.
Jun 19 14:41:50 dist-test-slave-32hj krb5kdc[22651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880109, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 14:41:50.112170 22882 log.cc:824] T 00000000000000000000000000000000 P f78f6d1d2a964a0d9828228d39d29619: Log is configured to *not* fsync() on all Append() calls
I20260619 14:41:50.111109 22706 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "43ed6e24011642cd988958cde97c58da"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46906
I20260619 14:41:50.113754 22732 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42439 } attempt: 3
I20260619 14:41:50.114729 22882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f78f6d1d2a964a0d9828228d39d29619: No bootstrap required, opened a new log
I20260619 14:41:50.115590 22810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:50.107175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42439 (local address 127.0.0.1:55744)
0619 14:41:50.107291 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:50.107306 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:50.107376 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:50.107623 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:50.107626 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:50.107786 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:41:50.107880 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:50.107887 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:50.108764 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:50.108767 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:50.109486 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:50.109496 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:50.110056 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:50.110060 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:50.110140 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:50.110618 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:41:50.110631 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:41:50.112066 (+ 1435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:41:50.113937 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:41:50.113941 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:41:50.113943 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:41:50.114194 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:41:50.114489 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:41:50.114493 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:41:50.114495 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:41:50.114591 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:41:50.115071 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:41:50.115074 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:41:50.115190 (+ 116us) client_negotiation.cc:769] Sending connection context
0619 14:41:50.115294 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 14:41:50.115429 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 14:41:50.115609 22886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:50.107329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55744 (local address 127.0.98.60:42439)
0619 14:41:50.107432 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:50.107436 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:50.107448 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:41:50.107477 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:50.107480 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:50.107516 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:50.107591 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:50.107969 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:50.108597 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:50.109614 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:50.109941 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:50.110036 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:50.112225 (+ 2189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:50.112229 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:50.112232 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:41:50.112260 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:50.113813 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:50.114312 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:50.114315 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:50.114317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:50.114396 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:50.114689 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:50.114692 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:50.114694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:50.114933 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:50.115067 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:50.115354 (+ 287us) server_negotiation.cc:299] Negotiation successful
0619 14:41:50.115471 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":40}
I20260619 14:41:50.118444 22882 raft_consensus.cc:348] T 00000000000000000000000000000000 P f78f6d1d2a964a0d9828228d39d29619 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43ed6e24011642cd988958cde97c58da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38917 } } peers { permanent_uuid: "26ab6d6de7bf404c9f92af9f38d3252a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45343 } } peers { permanent_uuid: "f78f6d1d2a964a0d9828228d39d29619" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42439 } }
I20260619 14:41:50.118716 22882 raft_consensus.cc:729] T 00000000000000000000000000000000 P f78f6d1d2a964a0d9828228d39d29619 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f78f6d1d2a964a0d9828228d39d29619, State: Initialized, Role: FOLLOWER
I20260619 14:41:50.118808 22807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26ab6d6de7bf404c9f92af9f38d3252a: Bootstrap starting.
I20260619 14:41:50.119065 22882 consensus_queue.cc:260] T 00000000000000000000000000000000 P f78f6d1d2a964a0d9828228d39d29619 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43ed6e24011642cd988958cde97c58da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38917 } } peers { permanent_uuid: "26ab6d6de7bf404c9f92af9f38d3252a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45343 } } peers { permanent_uuid: "f78f6d1d2a964a0d9828228d39d29619" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42439 } }
Jun 19 14:41:50 dist-test-slave-32hj krb5kdc[22651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880109, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 14:41:50.119704 22889 sys_catalog.cc:455] T 00000000000000000000000000000000 P f78f6d1d2a964a0d9828228d39d29619 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43ed6e24011642cd988958cde97c58da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38917 } } peers { permanent_uuid: "26ab6d6de7bf404c9f92af9f38d3252a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45343 } } peers { permanent_uuid: "f78f6d1d2a964a0d9828228d39d29619" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42439 } } }
I20260619 14:41:50.120008 22889 sys_catalog.cc:458] T 00000000000000000000000000000000 P f78f6d1d2a964a0d9828228d39d29619 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:41:50.120479 22882 sys_catalog.cc:564] T 00000000000000000000000000000000 P f78f6d1d2a964a0d9828228d39d29619 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:41:50.121608 22807 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26ab6d6de7bf404c9f92af9f38d3252a: Neither blocks nor log segments found. Creating new log.
I20260619 14:41:50.122475 22807 log.cc:824] T 00000000000000000000000000000000 P 26ab6d6de7bf404c9f92af9f38d3252a: Log is configured to *not* fsync() on all Append() calls
I20260619 14:41:50.123577 22734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:50.114134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42439 (local address 127.0.0.1:55754)
0619 14:41:50.114312 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:50.114326 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:50.114404 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:50.114862 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:50.114865 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:50.115039 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:41:50.115150 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:50.115157 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:50.116204 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:50.116208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:50.117043 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:50.117055 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:50.117845 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:50.117849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:50.117939 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:50.118399 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:41:50.118413 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:41:50.119822 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:41:50.121879 (+ 2057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:41:50.121883 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:41:50.121885 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:41:50.122127 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:41:50.122446 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:41:50.122449 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:41:50.122451 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:41:50.122547 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:41:50.123078 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:41:50.123081 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:41:50.123195 (+ 114us) client_negotiation.cc:769] Sending connection context
0619 14:41:50.123300 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 14:41:50.123449 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104}
I20260619 14:41:50.124464 22888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:50.114184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55754 (local address 127.0.98.60:42439)
0619 14:41:50.114626 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:50.114635 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:50.114648 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:41:50.114691 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:50.114693 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:50.114732 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:50.115258 (+ 526us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:50.115359 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:50.116088 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:50.117209 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:50.117716 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:50.117871 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:50.119987 (+ 2116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:50.119991 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:50.119994 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:41:50.120017 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:50.121772 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:50.122238 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:50.122242 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:50.122244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:50.122332 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:50.122644 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:50.122647 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:50.122649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:50.122902 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:50.123003 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:50.124176 (+ 1173us) server_negotiation.cc:299] Negotiation successful
0619 14:41:50.124298 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":96,"threads_started":1}
I20260619 14:41:50.124782 22807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26ab6d6de7bf404c9f92af9f38d3252a: No bootstrap required, opened a new log
W20260619 14:41:50.126104 22900 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f78f6d1d2a964a0d9828228d39d29619: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:41:50.126185 22900 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:41:50.128674 22807 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26ab6d6de7bf404c9f92af9f38d3252a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43ed6e24011642cd988958cde97c58da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38917 } } peers { permanent_uuid: "26ab6d6de7bf404c9f92af9f38d3252a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45343 } } peers { permanent_uuid: "f78f6d1d2a964a0d9828228d39d29619" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42439 } }
I20260619 14:41:50.125237 22813 master_runner.cc:186] Error getting master registration for 127.0.98.62:38917: OK, instance_id { permanent_uuid: "43ed6e24011642cd988958cde97c58da" instance_seqno: 1781880109653138 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 14:41:50.128912 22807 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26ab6d6de7bf404c9f92af9f38d3252a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26ab6d6de7bf404c9f92af9f38d3252a, State: Initialized, Role: FOLLOWER
I20260619 14:41:50.128994 22813 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 14:41:50.129244 22807 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26ab6d6de7bf404c9f92af9f38d3252a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43ed6e24011642cd988958cde97c58da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38917 } } peers { permanent_uuid: "26ab6d6de7bf404c9f92af9f38d3252a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45343 } } peers { permanent_uuid: "f78f6d1d2a964a0d9828228d39d29619" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42439 } }
I20260619 14:41:50.132026 22732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43ed6e24011642cd988958cde97c58da: Bootstrap starting.
I20260619 14:41:50.134788 22732 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43ed6e24011642cd988958cde97c58da: Neither blocks nor log segments found. Creating new log.
I20260619 14:41:50.135105 22903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26ab6d6de7bf404c9f92af9f38d3252a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43ed6e24011642cd988958cde97c58da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38917 } } peers { permanent_uuid: "26ab6d6de7bf404c9f92af9f38d3252a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45343 } } peers { permanent_uuid: "f78f6d1d2a964a0d9828228d39d29619" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42439 } } }
I20260619 14:41:50.135424 22903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26ab6d6de7bf404c9f92af9f38d3252a [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:41:50.135778 22732 log.cc:824] T 00000000000000000000000000000000 P 43ed6e24011642cd988958cde97c58da: Log is configured to *not* fsync() on all Append() calls
I20260619 14:41:50.135941 22807 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26ab6d6de7bf404c9f92af9f38d3252a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:41:50.137897 22732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43ed6e24011642cd988958cde97c58da: No bootstrap required, opened a new log
I20260619 14:41:50.140976 22732 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43ed6e24011642cd988958cde97c58da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43ed6e24011642cd988958cde97c58da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38917 } } peers { permanent_uuid: "26ab6d6de7bf404c9f92af9f38d3252a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45343 } } peers { permanent_uuid: "f78f6d1d2a964a0d9828228d39d29619" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42439 } }
I20260619 14:41:50.141196 22732 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43ed6e24011642cd988958cde97c58da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43ed6e24011642cd988958cde97c58da, State: Initialized, Role: FOLLOWER
W20260619 14:41:50.141417 22915 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 26ab6d6de7bf404c9f92af9f38d3252a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:41:50.141525 22915 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:41:50.141516 22732 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43ed6e24011642cd988958cde97c58da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43ed6e24011642cd988958cde97c58da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38917 } } peers { permanent_uuid: "26ab6d6de7bf404c9f92af9f38d3252a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45343 } } peers { permanent_uuid: "f78f6d1d2a964a0d9828228d39d29619" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42439 } }
I20260619 14:41:50.142786 22732 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43ed6e24011642cd988958cde97c58da [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:41:50.142874 22706 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43ed6e24011642cd988958cde97c58da [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 14:41:50.143009 22706 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43ed6e24011642cd988958cde97c58da [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:41:50.144374 22706 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43ed6e24011642cd988958cde97c58da [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43ed6e24011642cd988958cde97c58da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38917 } } peers { permanent_uuid: "26ab6d6de7bf404c9f92af9f38d3252a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45343 } } peers { permanent_uuid: "f78f6d1d2a964a0d9828228d39d29619" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42439 } }
I20260619 14:41:50.145326 22706 leader_election.cc:288] T 00000000000000000000000000000000 P 43ed6e24011642cd988958cde97c58da [CANDIDATE]: Term 1 election: Requested vote from peers 26ab6d6de7bf404c9f92af9f38d3252a (127.0.98.61:45343), f78f6d1d2a964a0d9828228d39d29619 (127.0.98.60:42439)
I20260619 14:41:50.145856 22781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "43ed6e24011642cd988958cde97c58da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "26ab6d6de7bf404c9f92af9f38d3252a"
I20260619 14:41:50.146062 22781 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26ab6d6de7bf404c9f92af9f38d3252a [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:41:50.147318 22781 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 26ab6d6de7bf404c9f92af9f38d3252a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43ed6e24011642cd988958cde97c58da in term 1.
I20260619 14:41:50.147917 22856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "43ed6e24011642cd988958cde97c58da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f78f6d1d2a964a0d9828228d39d29619"
I20260619 14:41:50.148168 22856 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f78f6d1d2a964a0d9828228d39d29619 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:41:50.149389 22856 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f78f6d1d2a964a0d9828228d39d29619 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43ed6e24011642cd988958cde97c58da in term 1.
W20260619 14:41:50.150086 22685 tablet.cc:2369] T 00000000000000000000000000000000 P 43ed6e24011642cd988958cde97c58da: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:41:50.153002 22682 leader_election.cc:302] T 00000000000000000000000000000000 P 43ed6e24011642cd988958cde97c58da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43ed6e24011642cd988958cde97c58da, f78f6d1d2a964a0d9828228d39d29619; no voters:
I20260619 14:41:50.153573 22933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43ed6e24011642cd988958cde97c58da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43ed6e24011642cd988958cde97c58da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38917 } } peers { permanent_uuid: "26ab6d6de7bf404c9f92af9f38d3252a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45343 } } peers { permanent_uuid: "f78f6d1d2a964a0d9828228d39d29619" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42439 } } }
I20260619 14:41:50.153825 22933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43ed6e24011642cd988958cde97c58da [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:41:50.153995 22933 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43ed6e24011642cd988958cde97c58da [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:41:50.154121 22933 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43ed6e24011642cd988958cde97c58da [term 1 LEADER]: Becoming Leader. State: Replica: 43ed6e24011642cd988958cde97c58da, State: Running, Role: LEADER
I20260619 14:41:50.154520 22933 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43ed6e24011642cd988958cde97c58da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43ed6e24011642cd988958cde97c58da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38917 } } peers { permanent_uuid: "26ab6d6de7bf404c9f92af9f38d3252a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45343 } } peers { permanent_uuid: "f78f6d1d2a964a0d9828228d39d29619" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42439 } }
I20260619 14:41:50.149941 22739 master_runner.cc:186] Error getting master registration for 127.0.98.62:38917: OK, instance_id { permanent_uuid: "43ed6e24011642cd988958cde97c58da" instance_seqno: 1781880109653138 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 14:41:50.155231 22739 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 14:41:50.157063 22933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43ed6e24011642cd988958cde97c58da [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43ed6e24011642cd988958cde97c58da. Latest consensus state: current_term: 1 leader_uuid: "43ed6e24011642cd988958cde97c58da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43ed6e24011642cd988958cde97c58da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38917 } } peers { permanent_uuid: "26ab6d6de7bf404c9f92af9f38d3252a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45343 } } peers { permanent_uuid: "f78f6d1d2a964a0d9828228d39d29619" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42439 } } }
I20260619 14:41:50.157246 22933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43ed6e24011642cd988958cde97c58da [sys.catalog]: This master's current role is: LEADER
W20260619 14:41:50.165365 22932 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 43ed6e24011642cd988958cde97c58da: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:41:50.165462 22932 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:41:50.178491 22936 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:41:50.179517 22936 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:41:50.181239 22888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:50.159439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55764 (local address 127.0.98.60:42439)
0619 14:41:50.159552 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:50.159555 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:50.168958 (+ 9403us) server_negotiation.cc:407] Connection header received
0619 14:41:50.169126 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:50.169129 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:50.169189 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:50.169236 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:50.170121 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:50.170865 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:50.172657 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:50.173022 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:50.173122 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:50.176136 (+ 3014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:50.176140 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:50.176143 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:41:50.176172 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:50.177934 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:50.179796 (+ 1862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:50.179800 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:50.179802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:50.179894 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:50.180211 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:50.180214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:50.180215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:50.180432 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:50.180547 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:50.180937 (+ 390us) server_negotiation.cc:299] Negotiation successful
0619 14:41:50.181064 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 14:41:50.182257 22812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:50.157012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37630 (local address 127.0.98.61:45343)
0619 14:41:50.157649 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:50.157654 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:50.166980 (+ 9326us) server_negotiation.cc:407] Connection header received
0619 14:41:50.167162 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:50.167167 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:50.167241 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:50.167292 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:50.170008 (+ 2716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:50.170781 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:50.171852 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:50.172180 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:50.172249 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:50.175777 (+ 3528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:50.175781 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:50.175784 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:41:50.175813 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:50.178768 (+ 2955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:50.180984 (+ 2216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:50.180988 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:50.180989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:50.181071 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:50.181347 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:50.181350 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:50.181352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:50.181557 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:50.181675 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:50.181999 (+ 324us) server_negotiation.cc:299] Negotiation successful
0619 14:41:50.182113 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555}
I20260619 14:41:50.183350 22735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:50.156299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46918 (local address 127.0.98.62:38917)
0619 14:41:50.156415 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:50.156419 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:50.162887 (+ 6468us) server_negotiation.cc:407] Connection header received
0619 14:41:50.165719 (+ 2832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:50.165723 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:50.165768 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:50.165815 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:50.168194 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:50.168976 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:50.171845 (+ 2869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:50.172180 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:50.172249 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:50.174873 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:50.174877 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:50.174880 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:41:50.174910 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:50.179333 (+ 4423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:50.182050 (+ 2717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:50.182053 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:50.182055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:50.182151 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:50.182440 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:50.182444 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:50.182446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:50.182654 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:50.182773 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:50.183090 (+ 317us) server_negotiation.cc:299] Negotiation successful
0619 14:41:50.183205 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 14:41:50.196273 22856 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f78f6d1d2a964a0d9828228d39d29619 [term 1 FOLLOWER]: Refusing update from remote peer 43ed6e24011642cd988958cde97c58da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:41:50.198689 22917 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 43ed6e24011642cd988958cde97c58da [LEADER]: Connected to new peer: Peer: permanent_uuid: "f78f6d1d2a964a0d9828228d39d29619" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:41:50.203299 22889 sys_catalog.cc:455] T 00000000000000000000000000000000 P f78f6d1d2a964a0d9828228d39d29619 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43ed6e24011642cd988958cde97c58da. Latest consensus state: current_term: 1 leader_uuid: "43ed6e24011642cd988958cde97c58da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43ed6e24011642cd988958cde97c58da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38917 } } peers { permanent_uuid: "26ab6d6de7bf404c9f92af9f38d3252a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45343 } } peers { permanent_uuid: "f78f6d1d2a964a0d9828228d39d29619" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42439 } } }
I20260619 14:41:50.203436 22889 sys_catalog.cc:458] T 00000000000000000000000000000000 P f78f6d1d2a964a0d9828228d39d29619 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:41:50.203778 22781 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 26ab6d6de7bf404c9f92af9f38d3252a [term 1 FOLLOWER]: Refusing update from remote peer 43ed6e24011642cd988958cde97c58da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:41:50.204372 22917 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 43ed6e24011642cd988958cde97c58da [LEADER]: Connected to new peer: Peer: permanent_uuid: "26ab6d6de7bf404c9f92af9f38d3252a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:41:50.208488 22903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26ab6d6de7bf404c9f92af9f38d3252a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43ed6e24011642cd988958cde97c58da. Latest consensus state: current_term: 1 leader_uuid: "43ed6e24011642cd988958cde97c58da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43ed6e24011642cd988958cde97c58da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38917 } } peers { permanent_uuid: "26ab6d6de7bf404c9f92af9f38d3252a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45343 } } peers { permanent_uuid: "f78f6d1d2a964a0d9828228d39d29619" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42439 } } }
I20260619 14:41:50.208617 22903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26ab6d6de7bf404c9f92af9f38d3252a [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:41:50.213857 22917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43ed6e24011642cd988958cde97c58da [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "43ed6e24011642cd988958cde97c58da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43ed6e24011642cd988958cde97c58da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38917 } } peers { permanent_uuid: "26ab6d6de7bf404c9f92af9f38d3252a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45343 } } peers { permanent_uuid: "f78f6d1d2a964a0d9828228d39d29619" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42439 } } }
I20260619 14:41:50.214015 22917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43ed6e24011642cd988958cde97c58da [sys.catalog]: This master's current role is: LEADER
I20260619 14:41:50.215250 22941 mvcc.cc:205] Tried to move back new op lower bound from 7298580931356950528 to 7298580931199107072. Current Snapshot: MvccSnapshot[applied={T|T < 7298580931356950528}]
I20260619 14:41:50.215407 22889 sys_catalog.cc:455] T 00000000000000000000000000000000 P f78f6d1d2a964a0d9828228d39d29619 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "43ed6e24011642cd988958cde97c58da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43ed6e24011642cd988958cde97c58da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38917 } } peers { permanent_uuid: "26ab6d6de7bf404c9f92af9f38d3252a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45343 } } peers { permanent_uuid: "f78f6d1d2a964a0d9828228d39d29619" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42439 } } }
I20260619 14:41:50.215508 22889 sys_catalog.cc:458] T 00000000000000000000000000000000 P f78f6d1d2a964a0d9828228d39d29619 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:41:50.215857 22940 mvcc.cc:205] Tried to move back new op lower bound from 7298580931356950528 to 7298580931199107072. Current Snapshot: MvccSnapshot[applied={T|T < 7298580931356950528}]
I20260619 14:41:50.216634 22933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43ed6e24011642cd988958cde97c58da [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "43ed6e24011642cd988958cde97c58da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43ed6e24011642cd988958cde97c58da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38917 } } peers { permanent_uuid: "26ab6d6de7bf404c9f92af9f38d3252a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45343 } } peers { permanent_uuid: "f78f6d1d2a964a0d9828228d39d29619" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42439 } } }
I20260619 14:41:50.216768 22933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43ed6e24011642cd988958cde97c58da [sys.catalog]: This master's current role is: LEADER
I20260619 14:41:50.217810 22903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26ab6d6de7bf404c9f92af9f38d3252a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "43ed6e24011642cd988958cde97c58da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43ed6e24011642cd988958cde97c58da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38917 } } peers { permanent_uuid: "26ab6d6de7bf404c9f92af9f38d3252a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45343 } } peers { permanent_uuid: "f78f6d1d2a964a0d9828228d39d29619" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42439 } } }
I20260619 14:41:50.217928 22903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26ab6d6de7bf404c9f92af9f38d3252a [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:41:50.220760 22943 mvcc.cc:205] Tried to move back new op lower bound from 7298580931356950528 to 7298580931199107072. Current Snapshot: MvccSnapshot[applied={T|T < 7298580931356950528}]
I20260619 14:41:50.222368 22936 catalog_manager.cc:1347] Generated new cluster ID: 65dec696be38495c8aaa72c7c8e6029f
I20260619 14:41:50.222438 22936 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:41:50.253633 22936 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:41:50.254931 22936 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:41:50.272073 22936 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 43ed6e24011642cd988958cde97c58da: Generated new TSK 0
I20260619 14:41:50.272953 22936 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 14:41:50.304203 22735 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:41:50.300755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46926 (local address 127.0.98.62:38917)
0619 14:41:50.300893 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:50.300898 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:50.302840 (+ 1942us) server_negotiation.cc:407] Connection header received
0619 14:41:50.303052 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:50.303056 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:50.303113 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:50.303170 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:50.304063 (+ 893us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46926: BlockingRecv error: recv got EOF from 127.0.0.1:46926 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260619 14:41:50.305438 22812 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:41:50.301566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37640 (local address 127.0.98.61:45343)
0619 14:41:50.301707 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:50.301711 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:50.303453 (+ 1742us) server_negotiation.cc:407] Connection header received
0619 14:41:50.304633 (+ 1180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:50.304636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:50.304688 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:50.304766 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:50.305320 (+ 554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37640: BlockingRecv error: recv got EOF from 127.0.0.1:37640 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260619 14:41:50.307106 22888 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:41:50.302197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55778 (local address 127.0.98.60:42439)
0619 14:41:50.302311 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:50.302315 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:50.305934 (+ 3619us) server_negotiation.cc:407] Connection header received
0619 14:41:50.306353 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:50.306356 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:50.306408 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:50.306461 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:50.306968 (+ 507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55778: BlockingRecv error: recv got EOF from 127.0.0.1:55778 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260619 14:41:50.312875 22812 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:41:50.310249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37644 (local address 127.0.98.61:45343)
0619 14:41:50.311223 (+ 974us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:50.311227 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:50.311790 (+ 563us) server_negotiation.cc:407] Connection header received
0619 14:41:50.312010 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:50.312013 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:50.312095 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:50.312152 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:50.312681 (+ 529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37644: BlockingRecv error: recv got EOF from 127.0.0.1:37644 (error 108)
Metrics: {"server-negotiator.queue_time_us":873}
W20260619 14:41:50.320123 22812 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:41:50.318570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37646 (local address 127.0.98.61:45343)
0619 14:41:50.318685 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:50.318689 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:50.319206 (+ 517us) server_negotiation.cc:407] Connection header received
0619 14:41:50.319389 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:50.319393 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:50.319442 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:50.319489 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:50.320010 (+ 521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37646: BlockingRecv error: recv got EOF from 127.0.0.1:37646 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260619 14:41:50.325865 22812 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:41:50.324335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37658 (local address 127.0.98.61:45343)
0619 14:41:50.324430 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:50.324434 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:50.325031 (+ 597us) server_negotiation.cc:407] Connection header received
0619 14:41:50.325205 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:50.325208 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:50.325255 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:50.325300 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:50.325762 (+ 462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37658: BlockingRecv error: recv got EOF from 127.0.0.1:37658 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260619 14:41:50.333225 22812 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:41:50.331764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37664 (local address 127.0.98.61:45343)
0619 14:41:50.331886 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:50.331891 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:50.332356 (+ 465us) server_negotiation.cc:407] Connection header received
0619 14:41:50.332522 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:50.332525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:50.332572 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:50.332630 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:50.333120 (+ 490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37664: BlockingRecv error: recv got EOF from 127.0.0.1:37664 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 14:41:50.344561 22812 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:41:50.343251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37666 (local address 127.0.98.61:45343)
0619 14:41:50.343398 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:50.343403 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:50.343671 (+ 268us) server_negotiation.cc:407] Connection header received
0619 14:41:50.343801 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:50.343805 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:50.343892 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:50.343965 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:50.344423 (+ 458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37666: BlockingRecv error: recv got EOF from 127.0.0.1:37666 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260619 14:41:50.363969 22812 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:41:50.362865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37668 (local address 127.0.98.61:45343)
0619 14:41:50.363001 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:50.363006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:50.363098 (+ 92us) server_negotiation.cc:407] Connection header received
0619 14:41:50.363225 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:50.363228 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:50.363283 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:50.363382 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:50.363848 (+ 466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37668: BlockingRecv error: recv got EOF from 127.0.0.1:37668 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260619 14:41:50.399231 22812 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:41:50.398204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37678 (local address 127.0.98.61:45343)
0619 14:41:50.398369 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:50.398373 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:50.398456 (+ 83us) server_negotiation.cc:407] Connection header received
0619 14:41:50.398579 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:50.398582 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:50.398628 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:50.398707 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:50.399084 (+ 377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37678: BlockingRecv error: recv got EOF from 127.0.0.1:37678 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260619 14:41:50.466538 22812 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:41:50.465411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37694 (local address 127.0.98.61:45343)
0619 14:41:50.465582 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:50.465587 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:50.465684 (+ 97us) server_negotiation.cc:407] Connection header received
0619 14:41:50.465832 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:50.465836 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:50.465883 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:50.465950 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:50.466410 (+ 460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37694: BlockingRecv error: recv got EOF from 127.0.0.1:37694 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260619 14:41:50.597680 22812 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:41:50.596598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37700 (local address 127.0.98.61:45343)
0619 14:41:50.596768 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:50.596773 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:50.596870 (+ 97us) server_negotiation.cc:407] Connection header received
0619 14:41:50.597014 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:50.597017 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:50.597062 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:50.597146 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:50.597563 (+ 417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37700: BlockingRecv error: recv got EOF from 127.0.0.1:37700 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260619 14:41:50.856839 22812 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:41:50.855692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37708 (local address 127.0.98.61:45343)
0619 14:41:50.855828 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:50.855833 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:50.855926 (+ 93us) server_negotiation.cc:407] Connection header received
0619 14:41:50.856078 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:50.856082 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:50.856129 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:50.856202 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:50.856669 (+ 467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37708: BlockingRecv error: recv got EOF from 127.0.0.1:37708 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 14:41:51.127504 22900 catalog_manager.cc:1259] Loaded cluster ID: 65dec696be38495c8aaa72c7c8e6029f
I20260619 14:41:51.127599 22900 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f78f6d1d2a964a0d9828228d39d29619: loading cluster ID for follower catalog manager: success
I20260619 14:41:51.129329 22900 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f78f6d1d2a964a0d9828228d39d29619: acquiring CA information for follower catalog manager: success
I20260619 14:41:51.130542 22900 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f78f6d1d2a964a0d9828228d39d29619: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 14:41:51.142982 22915 catalog_manager.cc:1259] Loaded cluster ID: 65dec696be38495c8aaa72c7c8e6029f
I20260619 14:41:51.143056 22915 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 26ab6d6de7bf404c9f92af9f38d3252a: loading cluster ID for follower catalog manager: success
I20260619 14:41:51.144889 22915 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 26ab6d6de7bf404c9f92af9f38d3252a: acquiring CA information for follower catalog manager: success
I20260619 14:41:51.146066 22915 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 26ab6d6de7bf404c9f92af9f38d3252a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 14:41:51.374486 23013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:51.370828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37712 (local address 127.0.98.61:45343)
0619 14:41:51.371094 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:51.371097 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:51.371113 (+ 16us) server_negotiation.cc:407] Connection header received
0619 14:41:51.371268 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:51.371271 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:51.371397 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:51.371498 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:41:51.371779 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:51.372297 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:51.373145 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:51.373370 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:51.373451 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:51.373708 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:41:51.374141 (+ 433us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:41:51.374229 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:51.374271 (+ 42us) server_negotiation.cc:299] Negotiation successful
0619 14:41:51.374320 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":83,"threads_started":1}
I20260619 14:41:51.380560 23019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:51.377304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55788 (local address 127.0.98.60:42439)
0619 14:41:51.377531 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:51.377534 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:51.377624 (+ 90us) server_negotiation.cc:407] Connection header received
0619 14:41:51.377725 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:51.377727 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:51.377835 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:51.377924 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:41:51.378067 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:51.378556 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:51.379299 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:51.379502 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:51.379567 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:51.379773 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:41:51.380173 (+ 400us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:41:51.380261 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:51.380365 (+ 104us) server_negotiation.cc:299] Negotiation successful
0619 14:41:51.380409 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":76,"threads_started":1}
I20260619 14:41:51.387382 23013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:51.383775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37714 (local address 127.0.98.61:45343)
0619 14:41:51.383909 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:51.383913 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:51.384054 (+ 141us) server_negotiation.cc:407] Connection header received
0619 14:41:51.384264 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:51.384267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:51.384319 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:51.384368 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:41:51.384703 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:51.385303 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:51.386322 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:51.386544 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:51.386630 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:51.386893 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:41:51.386971 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:41:51.387012 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:51.387205 (+ 193us) server_negotiation.cc:299] Negotiation successful
0619 14:41:51.387258 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 14:41:51.387928 23019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:51.384244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55790 (local address 127.0.98.60:42439)
0619 14:41:51.384345 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:51.384348 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:51.384400 (+ 52us) server_negotiation.cc:407] Connection header received
0619 14:41:51.384572 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:51.384575 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:51.384622 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:51.384689 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:41:51.384968 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:51.385667 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:51.386752 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:51.387050 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:51.387139 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:51.387373 (+ 234us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:41:51.387447 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:41:51.387511 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:51.387750 (+ 239us) server_negotiation.cc:299] Negotiation successful
0619 14:41:51.387803 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 14:41:51.388151 23025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:51.383507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46928 (local address 127.0.98.62:38917)
0619 14:41:51.383770 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:51.383774 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:51.383788 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:41:51.383882 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:51.383885 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:51.384031 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:51.384154 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:41:51.384476 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:51.385227 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:51.386360 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:51.386706 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:51.386793 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:51.387133 (+ 340us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:41:51.387476 (+ 343us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:41:51.387559 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:51.387960 (+ 401us) server_negotiation.cc:299] Negotiation successful
0619 14:41:51.388021 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":80,"threads_started":1}
I20260619 14:41:51.391304 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 22664
I20260619 14:41:51.396847 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 22739
I20260619 14:41:51.401659 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 22813
2026-06-19T14:41:51Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2997 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:41:51 dist-test-slave-32hj krb5kdc[23030](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:41:51 dist-test-slave-32hj krb5kdc[23030](info): set up 2 sockets
Jun 19 14:41:51 dist-test-slave-32hj krb5kdc[23030](info): commencing operation
krb5kdc: starting...
W20260619 14:41:52.364830 392 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/krb5kdc/test-user.keytab.
Jun 19 14:41:52 dist-test-slave-32hj krb5kdc[23030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:41:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:41:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:41:52.483819 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40495
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40495,127.0.98.61:33015,127.0.98.60:39047
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:41:52.589247 23043 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 14:41:52.589531 23043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:41:52.589565 23043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:41:52.589627 23043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:41:52.593547 23043 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:41:52.593606 23043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:41:52.593632 23043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:41:52.593654 23043 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:41:52.593683 23043 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:41:52.598822 23043 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40495,127.0.98.61:33015,127.0.98.60:39047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23043
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:41:52.599817 23043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:41:52.600826 23043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:41:52.608507 23051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:52.608507 23049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:52.608534 23048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:52.609431 23043 server_base.cc:1034] running on GCE node
I20260619 14:41:52.609900 23043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:41:52.610870 23043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:41:52.612013 23043 hybrid_clock.cc:648] HybridClock initialized: now 1781880112611981 us; error 49 us; skew 500 ppm
Jun 19 14:41:52 dist-test-slave-32hj krb5kdc[23030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880112, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:41:52.614566 23043 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:41:52.615386 23043 webserver.cc:460] Webserver started at http://127.0.98.62:41775/ using document root <none> and password file <none>
I20260619 14:41:52.615878 23043 fs_manager.cc:359] Metadata directory not provided
I20260619 14:41:52.615939 23043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:41:52.616127 23043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:41:52.618053 23043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/minicluster-data/master-0/data/instance:
uuid: "170742111f2f4017a44c972bdbd2e3bb"
format_stamp: "Formatted at 2026-06-19 14:41:52 on dist-test-slave-32hj"
server_key: "440657f0a05a42e24e93614523196d3f"
server_key_iv: "fe6ea1b622c5354e2dc45b87bf55de21"
server_key_version: "encryptionkey@0"
I20260619 14:41:52.618547 23043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/minicluster-data/master-0/wal/instance:
uuid: "170742111f2f4017a44c972bdbd2e3bb"
format_stamp: "Formatted at 2026-06-19 14:41:52 on dist-test-slave-32hj"
server_key: "440657f0a05a42e24e93614523196d3f"
server_key_iv: "fe6ea1b622c5354e2dc45b87bf55de21"
server_key_version: "encryptionkey@0"
I20260619 14:41:52.621969 23043 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:41:52.624317 23058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:52.625541 23043 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 14:41:52.625656 23043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/minicluster-data/master-0/wal
uuid: "170742111f2f4017a44c972bdbd2e3bb"
format_stamp: "Formatted at 2026-06-19 14:41:52 on dist-test-slave-32hj"
server_key: "440657f0a05a42e24e93614523196d3f"
server_key_iv: "fe6ea1b622c5354e2dc45b87bf55de21"
server_key_version: "encryptionkey@0"
I20260619 14:41:52.625749 23043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:41:52.642311 23043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:41:52.645395 23043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:41:52.645571 23043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:41:52.652401 23043 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40495
I20260619 14:41:52.652683 23110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40495 every 8 connection(s)
I20260619 14:41:52.653378 23043 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:41:52.656318 23111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:41:52.659530 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 23043
I20260619 14:41:52.659653 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/minicluster-data/master-0/wal/instance
I20260619 14:41:52.659972 392 external_mini_cluster.cc:1428] Setting key 6e2c7dda8a7068c864b94b6f09334715
I20260619 14:41:52.660394 23111 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40495 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Jun 19 14:41:52 dist-test-slave-32hj krb5kdc[23030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880112, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:41:52.676168 23115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:52.662954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40495 (local address 127.0.0.1:36218)
0619 14:41:52.663551 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:52.663599 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:52.664409 (+ 810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:52.665227 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:52.665232 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:52.665546 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:41:52.665857 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:52.665870 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:52.667501 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:52.667504 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:52.668266 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:52.668275 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:52.668943 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:52.668946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:52.669048 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:52.669558 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:41:52.669574 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:41:52.671423 (+ 1849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:41:52.673411 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:41:52.673419 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:41:52.673428 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:41:52.673633 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:41:52.673918 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:41:52.673921 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:41:52.673923 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:41:52.674022 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:41:52.674556 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:41:52.674562 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:41:52.674688 (+ 126us) client_negotiation.cc:769] Sending connection context
0619 14:41:52.674830 (+ 142us) client_negotiation.cc:240] Negotiation successful
0619 14:41:52.675181 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"mutex_wait_us":90,"spinlock_wait_cycles":96512,"thread_start_us":161,"threads_started":1}
I20260619 14:41:52.676168 23114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:52.662954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36218 (local address 127.0.98.62:40495)
0619 14:41:52.663431 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:52.663449 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:52.663811 (+ 362us) server_negotiation.cc:407] Connection header received
0619 14:41:52.664753 (+ 942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:52.664760 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:52.665028 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:52.665242 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:52.666003 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:52.667382 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:52.668371 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:52.668807 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:52.668924 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:52.671551 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:52.671556 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:52.671566 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:41:52.671586 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:52.673321 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:52.673728 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:52.673732 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:52.673734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:52.673830 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:52.674099 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:52.674102 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:52.674104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:52.674446 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:52.674554 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:52.674901 (+ 347us) server_negotiation.cc:299] Negotiation successful
0619 14:41:52.675114 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":317,"thread_start_us":146,"threads_started":1}
I20260619 14:41:52.678670 23111 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33015 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:41:52.679426 23115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:52.679143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33015 (local address 127.0.0.1:45792)
0619 14:41:52.679277 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:52.679347 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260619 14:41:52.679602 23059 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33015: connect: Connection refused (error 111)
I20260619 14:41:52.680464 23115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:52.680253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33015 (local address 127.0.0.1:45800)
0619 14:41:52.680365 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:52.680411 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 14:41:52.680584 23111 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33015: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33015: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260619 14:41:52.686026 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:33015
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40495,127.0.98.61:33015,127.0.98.60:39047
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:41:52.729825 23111 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33015 } attempt: 1
I20260619 14:41:52.730660 23115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:52.730378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33015 (local address 127.0.0.1:45810)
0619 14:41:52.730515 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:52.730574 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 14:41:52.731386 23115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:52.731179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33015 (local address 127.0.0.1:45824)
0619 14:41:52.731276 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:52.731329 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260619 14:41:52.731498 23111 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33015: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33015: connect: Connection refused (error 111)
W20260619 14:41:52.794291 23118 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 14:41:52.794575 23118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:41:52.794600 23118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:41:52.794653 23118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:41:52.798617 23118 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:41:52.798676 23118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:41:52.798693 23118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:41:52.798709 23118 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:41:52.798723 23118 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:41:52.799659 23111 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33015 } attempt: 2
I20260619 14:41:52.800530 23115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:52.800190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33015 (local address 127.0.0.1:45832)
0619 14:41:52.800363 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:52.800427 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260619 14:41:52.801213 23115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:52.801003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33015 (local address 127.0.0.1:45840)
0619 14:41:52.801102 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:52.801155 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260619 14:41:52.801321 23111 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33015: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33015: connect: Connection refused (error 111)
I20260619 14:41:52.804196 23118 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40495,127.0.98.61:33015,127.0.98.60:39047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:33015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:41:52.805241 23118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:41:52.806173 23118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:41:52.814311 23124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:52.814381 23126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:52.814318 23123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:52.814667 23118 server_base.cc:1034] running on GCE node
I20260619 14:41:52.815222 23118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:41:52.816334 23118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:41:52.817499 23118 hybrid_clock.cc:648] HybridClock initialized: now 1781880112817472 us; error 52 us; skew 500 ppm
Jun 19 14:41:52 dist-test-slave-32hj krb5kdc[23030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880112, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:41:52.820291 23118 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260619 14:41:52.821300 23118 webserver.cc:460] Webserver started at http://127.0.98.61:39999/ using document root <none> and password file <none>
I20260619 14:41:52.821843 23118 fs_manager.cc:359] Metadata directory not provided
I20260619 14:41:52.821909 23118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:41:52.822116 23118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:41:52.824177 23118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/minicluster-data/master-1/data/instance:
uuid: "c5a0a4fea8d447788d89d9093e36f4bb"
format_stamp: "Formatted at 2026-06-19 14:41:52 on dist-test-slave-32hj"
server_key: "a97c5bf158ad76aa0009eb0ab072721e"
server_key_iv: "a8cf6f00a559fe1a71a4a0d0d3dab2b4"
server_key_version: "encryptionkey@0"
I20260619 14:41:52.824759 23118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/minicluster-data/master-1/wal/instance:
uuid: "c5a0a4fea8d447788d89d9093e36f4bb"
format_stamp: "Formatted at 2026-06-19 14:41:52 on dist-test-slave-32hj"
server_key: "a97c5bf158ad76aa0009eb0ab072721e"
server_key_iv: "a8cf6f00a559fe1a71a4a0d0d3dab2b4"
server_key_version: "encryptionkey@0"
I20260619 14:41:52.828609 23118 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260619 14:41:52.831163 23133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:52.832479 23118 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:41:52.832643 23118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/minicluster-data/master-1/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/minicluster-data/master-1/wal
uuid: "c5a0a4fea8d447788d89d9093e36f4bb"
format_stamp: "Formatted at 2026-06-19 14:41:52 on dist-test-slave-32hj"
server_key: "a97c5bf158ad76aa0009eb0ab072721e"
server_key_iv: "a8cf6f00a559fe1a71a4a0d0d3dab2b4"
server_key_version: "encryptionkey@0"
I20260619 14:41:52.832774 23118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:41:52.843700 23118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:41:52.846767 23118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:41:52.846925 23118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:41:52.853264 23118 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:33015
I20260619 14:41:52.853516 23185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:33015 every 8 connection(s)
I20260619 14:41:52.854189 23118 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/minicluster-data/master-1/data/info.pb
I20260619 14:41:52.856918 23186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:41:52.860529 23186 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40495 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:41:52.862010 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 23118
I20260619 14:41:52.862134 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/minicluster-data/master-1/wal/instance
I20260619 14:41:52.862433 392 external_mini_cluster.cc:1428] Setting key 835671db72875c802a23c1209a585834
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 19 14:41:52 dist-test-slave-32hj krb5kdc[23030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880112, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:41:52.878202 23114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:52.862408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36224 (local address 127.0.98.62:40495)
0619 14:41:52.862565 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:52.862570 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:52.863536 (+ 966us) server_negotiation.cc:407] Connection header received
0619 14:41:52.864572 (+ 1036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:52.864577 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:52.864634 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:52.864761 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:52.866209 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:52.866782 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:52.868421 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:52.868650 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:52.868750 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:52.874199 (+ 5449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:52.874203 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:52.874206 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:41:52.874239 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:52.875957 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:52.876554 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:52.876556 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:52.876557 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:52.876634 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:52.876987 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:52.876989 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:52.876990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:52.877197 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:52.877295 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:52.877945 (+ 650us) server_negotiation.cc:299] Negotiation successful
0619 14:41:52.878056 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 14:41:52.879410 23188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:52.862847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40495 (local address 127.0.0.1:36224)
0619 14:41:52.863353 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:52.863408 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:52.864351 (+ 943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:52.864915 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:52.864924 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:52.865378 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:41:52.866035 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:52.866054 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:52.866917 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:52.866921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:52.868312 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:52.868324 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:52.868818 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:52.868822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:52.869019 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:52.869634 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:41:52.869653 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:41:52.873993 (+ 4340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:41:52.876097 (+ 2104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:41:52.876103 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:41:52.876121 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:41:52.876468 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:41:52.876710 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:41:52.876713 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:41:52.876715 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:41:52.876908 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:41:52.877309 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:41:52.877317 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:41:52.877684 (+ 367us) client_negotiation.cc:769] Sending connection context
0619 14:41:52.877904 (+ 220us) client_negotiation.cc:240] Negotiation successful
0619 14:41:52.878292 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":127,"threads_started":1}
I20260619 14:41:52.880789 23186 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33015 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:41:52 dist-test-slave-32hj krb5kdc[23030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880112, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260619 14:41:52.889706 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7PHJY_/build/debug/bin/kudu
/tmp/dist-test-task7PHJY_/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39047
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40495,127.0.98.61:33015,127.0.98.60:39047
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:41:52.889917 23188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:52.881308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33015 (local address 127.0.0.1:45842)
0619 14:41:52.881499 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:52.881513 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:52.881590 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:52.882130 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:52.882132 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:52.882268 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:41:52.882369 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:52.882374 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:52.883419 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:52.883421 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:52.883950 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:52.883959 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:52.884607 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:52.884612 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:52.884707 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:52.885219 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:41:52.885230 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:41:52.886454 (+ 1224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:41:52.888375 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:41:52.888378 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:41:52.888379 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:41:52.888564 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:41:52.888886 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:41:52.888889 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:41:52.888891 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:41:52.888972 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:41:52.889415 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:41:52.889418 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:41:52.889517 (+ 99us) client_negotiation.cc:769] Sending connection context
0619 14:41:52.889628 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 14:41:52.889760 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128}
I20260619 14:41:52.890048 23191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:52.881423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45842 (local address 127.0.98.61:33015)
0619 14:41:52.881781 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:52.881790 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:52.881810 (+ 20us) server_negotiation.cc:407] Connection header received
0619 14:41:52.881844 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:52.881848 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:52.881996 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:52.882107 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:52.882462 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:52.883290 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:52.884091 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:52.884462 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:52.884552 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:52.886616 (+ 2064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:52.886629 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:52.886646 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:41:52.886671 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:52.888263 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:52.888680 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:52.888686 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:52.888688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:52.888809 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:52.889049 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:52.889052 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:52.889054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:52.889320 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:52.889423 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:52.889714 (+ 291us) server_negotiation.cc:299] Negotiation successful
0619 14:41:52.889853 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":149,"threads_started":1}
I20260619 14:41:52.892428 23186 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39047 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:41:52.892464 23111 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33015 } attempt: 3
I20260619 14:41:52.893235 23188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:52.892912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39047 (local address 127.0.0.1:47900)
0619 14:41:52.893071 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:52.893159 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260619 14:41:52.893410 23135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39047: connect: Connection refused (error 111)
I20260619 14:41:52.894459 23188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:52.894270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39047 (local address 127.0.0.1:47904)
0619 14:41:52.894361 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:52.894406 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260619 14:41:52.894627 23186 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39047: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39047: connect: Connection refused (error 111)
Jun 19 14:41:52 dist-test-slave-32hj krb5kdc[23030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880112, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 14:41:52.901667 23115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:52.892883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33015 (local address 127.0.0.1:45852)
0619 14:41:52.893025 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:52.893041 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:52.893126 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:52.893392 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:52.893395 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:52.893570 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:41:52.893696 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:52.893704 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:52.894720 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:52.894725 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:52.895548 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:52.895559 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:52.896200 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:52.896203 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:52.896290 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:52.896867 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:41:52.896882 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:41:52.898381 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:41:52.900211 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:41:52.900214 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:41:52.900216 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:41:52.900402 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:41:52.900693 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:41:52.900695 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:41:52.900697 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:41:52.900831 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:41:52.901229 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:41:52.901232 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:41:52.901323 (+ 91us) client_negotiation.cc:769] Sending connection context
0619 14:41:52.901434 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 14:41:52.901545 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 14:41:52.901747 23191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:52.893054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45852 (local address 127.0.98.61:33015)
0619 14:41:52.893149 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:52.893153 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:52.893166 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:41:52.893197 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:52.893199 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:52.893237 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:52.893307 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:52.893799 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:52.894562 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:52.895709 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:52.896056 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:52.896160 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:52.898515 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:52.898519 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:52.898522 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:41:52.898549 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:52.900096 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:52.900486 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:52.900490 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:52.900492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:52.900586 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:52.900913 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:52.900916 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:52.900918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:52.901115 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:52.901205 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:52.901491 (+ 286us) server_negotiation.cc:299] Negotiation successful
0619 14:41:52.901607 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 14:41:52.902781 23111 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39047 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:41:52.903437 23115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:52.903222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39047 (local address 127.0.0.1:47918)
0619 14:41:52.903331 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:52.903377 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 14:41:52.904078 23115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:52.903848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39047 (local address 127.0.0.1:47928)
0619 14:41:52.903988 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:52.904028 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260619 14:41:52.904196 23111 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39047: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39047: connect: Connection refused (error 111)
I20260619 14:41:52.935360 23111 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39047 } attempt: 1
I20260619 14:41:52.936198 23115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:52.935910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39047 (local address 127.0.0.1:47934)
0619 14:41:52.936053 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:52.936116 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 14:41:52.937355 23115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:52.937104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39047 (local address 127.0.0.1:47944)
0619 14:41:52.937217 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:52.937265 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 14:41:52.937474 23111 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39047: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39047: connect: Connection refused (error 111)
I20260619 14:41:52.943882 23186 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39047 } attempt: 1
I20260619 14:41:52.944669 23188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:52.944382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39047 (local address 127.0.0.1:47952)
0619 14:41:52.944535 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:52.944593 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260619 14:41:52.945322 23188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:52.945119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39047 (local address 127.0.0.1:47966)
0619 14:41:52.945220 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:52.945263 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260619 14:41:52.945428 23186 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39047: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39047: connect: Connection refused (error 111)
W20260619 14:41:53.002573 23192 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 14:41:53.002949 23192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:41:53.002983 23192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:41:53.003038 23192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:41:53.008426 23192 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:41:53.008493 23192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:41:53.008518 23192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:41:53.008543 23192 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:41:53.008571 23192 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:41:53.012645 23111 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39047 } attempt: 2
I20260619 14:41:53.013453 23115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:53.013170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39047 (local address 127.0.0.1:47974)
0619 14:41:53.013308 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:53.013373 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 14:41:53.013598 23186 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39047 } attempt: 2
I20260619 14:41:53.014199 23115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:53.013959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39047 (local address 127.0.0.1:48004)
0619 14:41:53.014066 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:53.014129 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260619 14:41:53.014221 23188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:53.013961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39047 (local address 127.0.0.1:47988)
0619 14:41:53.014115 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:53.014161 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260619 14:41:53.014369 23111 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39047: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39047: connect: Connection refused (error 111)
I20260619 14:41:53.014648 23192 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40495,127.0.98.61:33015,127.0.98.60:39047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 13:59:48 UTC on ccycloud.quasar-kgtkto.root.comops.site
build id 4455765
I20260619 14:41:53.014832 23188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:53.014638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39047 (local address 127.0.0.1:48012)
0619 14:41:53.014732 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:53.014777 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260619 14:41:53.014974 23186 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39047: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39047: connect: Connection refused (error 111)
I20260619 14:41:53.015738 23192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:41:53.016784 23192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:41:53.024827 23198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:41:53.024911 23199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:53.025007 23192 server_base.cc:1034] running on GCE node
W20260619 14:41:53.024844 23201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:41:53.025694 23192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:41:53.026760 23192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:41:53.027915 23192 hybrid_clock.cc:648] HybridClock initialized: now 1781880113027894 us; error 34 us; skew 500 ppm
Jun 19 14:41:53 dist-test-slave-32hj krb5kdc[23030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880113, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:41:53.030572 23192 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260619 14:41:53.031409 23192 webserver.cc:460] Webserver started at http://127.0.98.60:33863/ using document root <none> and password file <none>
I20260619 14:41:53.031925 23192 fs_manager.cc:359] Metadata directory not provided
I20260619 14:41:53.031987 23192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:41:53.032189 23192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:41:53.034176 23192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/minicluster-data/master-2/data/instance:
uuid: "74d0d6c4050f45888d6e6eed817112e5"
format_stamp: "Formatted at 2026-06-19 14:41:53 on dist-test-slave-32hj"
server_key: "bbb925b4c171d02b12b853e3f3959129"
server_key_iv: "1dd63354825f17d0ad4e44a21827540e"
server_key_version: "encryptionkey@0"
I20260619 14:41:53.034696 23192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/minicluster-data/master-2/wal/instance:
uuid: "74d0d6c4050f45888d6e6eed817112e5"
format_stamp: "Formatted at 2026-06-19 14:41:53 on dist-test-slave-32hj"
server_key: "bbb925b4c171d02b12b853e3f3959129"
server_key_iv: "1dd63354825f17d0ad4e44a21827540e"
server_key_version: "encryptionkey@0"
I20260619 14:41:53.038228 23192 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:41:53.040669 23208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:41:53.042073 23192 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:41:53.042202 23192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/minicluster-data/master-2/data,/tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/minicluster-data/master-2/wal
uuid: "74d0d6c4050f45888d6e6eed817112e5"
format_stamp: "Formatted at 2026-06-19 14:41:53 on dist-test-slave-32hj"
server_key: "bbb925b4c171d02b12b853e3f3959129"
server_key_iv: "1dd63354825f17d0ad4e44a21827540e"
server_key_version: "encryptionkey@0"
I20260619 14:41:53.042303 23192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:41:53.058315 23192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:41:53.061524 23192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:41:53.061717 23192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:41:53.068595 23192 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:39047
I20260619 14:41:53.068888 23260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:39047 every 8 connection(s)
I20260619 14:41:53.069603 23192 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/minicluster-data/master-2/data/info.pb
I20260619 14:41:53.071350 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7PHJY_/build/debug/bin/kudu as pid 23192
I20260619 14:41:53.071484 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7PHJY_/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781879837648036-392-0/minicluster-data/master-2/wal/instance
I20260619 14:41:53.071856 392 external_mini_cluster.cc:1428] Setting key 91930f9eeb5bfa01389279c9d9bfbb03
I20260619 14:41:53.072858 23261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:41:53.076959 23261 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40495 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:41:53 dist-test-slave-32hj krb5kdc[23030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 19 14:41:53 dist-test-slave-32hj krb5kdc[23030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880113, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:41:53.086273 23264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:53.073222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48018 (local address 127.0.98.60:39047)
0619 14:41:53.073604 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:53.073621 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:53.073657 (+ 36us) server_negotiation.cc:407] Connection header received
0619 14:41:53.074326 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:53.074343 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:53.074679 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:53.074953 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:53.075823 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:53.076868 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:53.077985 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:53.078468 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:53.078632 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:53.081252 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:53.081257 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:53.081266 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:41:53.081292 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:53.083178 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:53.083666 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:53.083671 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:53.083675 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:53.083786 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:53.084059 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:53.084062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:53.084063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:53.084429 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:53.084601 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:53.084933 (+ 332us) server_negotiation.cc:299] Negotiation successful
0619 14:41:53.085222 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":110,"threads_started":1}
I20260619 14:41:53.087543 23265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:53.078835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40495 (local address 127.0.0.1:36230)
0619 14:41:53.079308 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:53.079349 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:53.079461 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:53.079722 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:53.079728 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:53.080058 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:41:53.080195 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:53.080203 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:53.080984 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:53.080988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:53.081682 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:53.081691 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:53.082087 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:53.082089 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:53.082183 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:53.082552 (+ 369us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:41:53.082566 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:41:53.084179 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:41:53.085901 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:41:53.085908 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:41:53.085912 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:41:53.086217 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:41:53.086496 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:41:53.086499 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:41:53.086501 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:41:53.086593 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:41:53.086984 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:41:53.086988 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:41:53.087096 (+ 108us) client_negotiation.cc:769] Sending connection context
0619 14:41:53.087217 (+ 121us) client_negotiation.cc:240] Negotiation successful
0619 14:41:53.087367 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":111,"threads_started":1}
I20260619 14:41:53.087569 23114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:53.078924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36230 (local address 127.0.98.62:40495)
0619 14:41:53.079067 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:53.079071 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:53.079428 (+ 357us) server_negotiation.cc:407] Connection header received
0619 14:41:53.079548 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:53.079552 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:53.079597 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:53.079663 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:53.080303 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:53.080863 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:53.081776 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:53.081984 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:53.082073 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:53.084295 (+ 2222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:53.084298 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:53.084300 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:41:53.084323 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:53.085785 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:53.086309 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:53.086313 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:53.086314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:53.086405 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:53.086673 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:53.086676 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:53.086678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:53.086889 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:53.086980 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:53.087310 (+ 330us) server_negotiation.cc:299] Negotiation successful
0619 14:41:53.087431 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 14:41:53.089167 23261 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33015 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:41:53 dist-test-slave-32hj krb5kdc[23030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 19 14:41:53 dist-test-slave-32hj krb5kdc[23030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880113, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 14:41:53.097756 23191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:53.089760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45866 (local address 127.0.98.61:33015)
0619 14:41:53.089900 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:53.089903 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:53.089916 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:41:53.089955 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:53.089957 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:53.089995 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:53.090061 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:53.090517 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:53.091320 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:53.092326 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:53.092559 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:53.092644 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:53.094917 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:53.094920 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:53.094922 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:41:53.094948 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:53.096266 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:53.096647 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:53.096651 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:53.096653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:53.096763 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:53.097006 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:53.097009 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:53.097011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:53.097188 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:53.097268 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:53.097512 (+ 244us) server_negotiation.cc:299] Negotiation successful
0619 14:41:53.097620 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 14:41:53.097751 23265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:53.089617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33015 (local address 127.0.0.1:45866)
0619 14:41:53.089756 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:53.089768 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:53.089831 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:53.090086 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:53.090089 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:53.090264 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:41:53.090370 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:53.090376 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:53.091467 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:53.091470 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:53.092215 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:53.092223 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:53.092657 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:53.092661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:53.092782 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:53.093295 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:41:53.093306 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:41:53.094777 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:41:53.096362 (+ 1585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:41:53.096366 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:41:53.096368 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:41:53.096557 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:41:53.096839 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:41:53.096842 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:41:53.096844 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:41:53.096922 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:41:53.097269 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:41:53.097272 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:41:53.097361 (+ 89us) client_negotiation.cc:769] Sending connection context
0619 14:41:53.097445 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 14:41:53.097578 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260619 14:41:53.097772 23114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:53.088847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36246 (local address 127.0.98.62:40495)
0619 14:41:53.088966 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:53.088971 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:53.088988 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:41:53.089018 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:53.089020 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:53.089061 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:53.089138 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:53.089813 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:53.090484 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:53.091530 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:53.091856 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:53.091967 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:53.094611 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:53.094614 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:53.094616 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:41:53.094640 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:53.096136 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:53.096594 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:53.096597 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:53.096599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:53.096680 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:53.096955 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:53.096958 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:53.096960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:53.097156 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:53.097378 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:53.097494 (+ 116us) server_negotiation.cc:299] Negotiation successful
0619 14:41:53.097620 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 14:41:53.098930 23261 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39047 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:41:53 dist-test-slave-32hj krb5kdc[23030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880113, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 19 14:41:53 dist-test-slave-32hj krb5kdc[23030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 14:41:53.106133 23186 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39047 } attempt: 3
I20260619 14:41:53.107129 23265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:53.099408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39047 (local address 127.0.0.1:48030)
0619 14:41:53.099501 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:53.099541 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:53.099608 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:53.099802 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:53.099804 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:53.099951 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:41:53.100047 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:53.100054 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:53.101011 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:53.101014 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:53.101800 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:53.101809 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:53.102307 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:53.102310 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:53.102367 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:53.102795 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:41:53.102805 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:41:53.104134 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:41:53.105725 (+ 1591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:41:53.105728 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:41:53.105730 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:41:53.105914 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:41:53.106148 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:41:53.106150 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:41:53.106152 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:41:53.106222 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:41:53.106639 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:41:53.106641 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:41:53.106722 (+ 81us) client_negotiation.cc:769] Sending connection context
0619 14:41:53.106824 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 14:41:53.106947 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260619 14:41:53.107474 23264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:53.099429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48030 (local address 127.0.98.60:39047)
0619 14:41:53.099532 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:53.099536 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:53.099577 (+ 41us) server_negotiation.cc:407] Connection header received
0619 14:41:53.099679 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:53.099680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:53.099714 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:53.099773 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:53.100138 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:53.100923 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:53.101902 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:53.102205 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:53.102291 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:53.104249 (+ 1958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:53.104253 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:53.104255 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:41:53.104274 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:53.105543 (+ 1269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:53.105994 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:53.105998 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:53.106000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:53.106073 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:53.106308 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:53.106311 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:53.106313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:53.106518 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:53.106606 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:53.107091 (+ 485us) server_negotiation.cc:299] Negotiation successful
0619 14:41:53.107337 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 14:41:53.107826 23191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:53.099203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45878 (local address 127.0.98.61:33015)
0619 14:41:53.099307 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:53.099311 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:53.099325 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:41:53.099467 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:53.099470 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:53.099514 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:53.099563 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:53.100335 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:53.101090 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:53.102033 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:53.102333 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:53.102415 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:53.104815 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:53.104818 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:53.104820 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:41:53.104840 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:53.106175 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:53.106669 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:53.106673 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:53.106675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:53.106756 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:53.107024 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:53.107028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:53.107030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:53.107247 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:53.107496 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:53.107568 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 14:41:53.107685 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 14:41:53.111284 23261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74d0d6c4050f45888d6e6eed817112e5: Bootstrap starting.
Jun 19 14:41:53 dist-test-slave-32hj krb5kdc[23030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880112, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 14:41:53.112190 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:41:53.113523 23111 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39047 } attempt: 3
I20260619 14:41:53.114271 23261 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74d0d6c4050f45888d6e6eed817112e5: Neither blocks nor log segments found. Creating new log.
I20260619 14:41:53.113555 23085 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "170742111f2f4017a44c972bdbd2e3bb"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36246
I20260619 14:41:53.115171 23261 log.cc:824] T 00000000000000000000000000000000 P 74d0d6c4050f45888d6e6eed817112e5: Log is configured to *not* fsync() on all Append() calls
I20260619 14:41:53.115458 23188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:53.106515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39047 (local address 127.0.0.1:48044)
0619 14:41:53.106620 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:53.106634 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:53.106704 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:53.107814 (+ 1110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:53.107818 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:53.107986 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:41:53.108109 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:53.108116 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:53.109126 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:53.109129 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:53.109727 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:53.109736 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:53.110139 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:53.110142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:53.110220 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:53.110686 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:41:53.110697 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:41:53.112081 (+ 1384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:41:53.113920 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:41:53.113924 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:41:53.113926 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:41:53.114161 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:41:53.114477 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:41:53.114480 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:41:53.114482 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:41:53.114574 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:41:53.115004 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:41:53.115007 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:41:53.115101 (+ 94us) client_negotiation.cc:769] Sending connection context
0619 14:41:53.115180 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 14:41:53.115309 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260619 14:41:53.115974 23264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:53.106700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48044 (local address 127.0.98.60:39047)
0619 14:41:53.107607 (+ 907us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:53.107611 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:53.107622 (+ 11us) server_negotiation.cc:407] Connection header received
0619 14:41:53.107650 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:53.107652 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:53.107690 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:53.108212 (+ 522us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:53.108290 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:53.108997 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:53.109839 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:53.110047 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:53.110122 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:53.112224 (+ 2102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:53.112228 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:53.112231 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:41:53.112254 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:53.113801 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:53.114285 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:53.114288 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:53.114290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:53.114369 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:53.114692 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:53.114695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:53.114696 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:53.114895 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:53.115638 (+ 743us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:53.115711 (+ 73us) server_negotiation.cc:299] Negotiation successful
0619 14:41:53.115820 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":842,"thread_start_us":177,"threads_started":1}
I20260619 14:41:53.118130 23261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74d0d6c4050f45888d6e6eed817112e5: No bootstrap required, opened a new log
I20260619 14:41:53.118818 23186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5a0a4fea8d447788d89d9093e36f4bb: Bootstrap starting.
Jun 19 14:41:53 dist-test-slave-32hj krb5kdc[23030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880112, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 14:41:53.121685 23186 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5a0a4fea8d447788d89d9093e36f4bb: Neither blocks nor log segments found. Creating new log.
I20260619 14:41:53.122184 23261 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74d0d6c4050f45888d6e6eed817112e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170742111f2f4017a44c972bdbd2e3bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40495 } } peers { permanent_uuid: "c5a0a4fea8d447788d89d9093e36f4bb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33015 } } peers { permanent_uuid: "74d0d6c4050f45888d6e6eed817112e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39047 } }
I20260619 14:41:53.122445 23261 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74d0d6c4050f45888d6e6eed817112e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74d0d6c4050f45888d6e6eed817112e5, State: Initialized, Role: FOLLOWER
I20260619 14:41:53.122601 23186 log.cc:824] T 00000000000000000000000000000000 P c5a0a4fea8d447788d89d9093e36f4bb: Log is configured to *not* fsync() on all Append() calls
I20260619 14:41:53.122789 23261 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74d0d6c4050f45888d6e6eed817112e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170742111f2f4017a44c972bdbd2e3bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40495 } } peers { permanent_uuid: "c5a0a4fea8d447788d89d9093e36f4bb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33015 } } peers { permanent_uuid: "74d0d6c4050f45888d6e6eed817112e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39047 } }
I20260619 14:41:53.123399 23269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74d0d6c4050f45888d6e6eed817112e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170742111f2f4017a44c972bdbd2e3bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40495 } } peers { permanent_uuid: "c5a0a4fea8d447788d89d9093e36f4bb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33015 } } peers { permanent_uuid: "74d0d6c4050f45888d6e6eed817112e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39047 } } }
I20260619 14:41:53.123602 23269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74d0d6c4050f45888d6e6eed817112e5 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:41:53.124050 23115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:53.113873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39047 (local address 127.0.0.1:48058)
0619 14:41:53.114106 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0619 14:41:53.114119 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:41:53.114191 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:41:53.115199 (+ 1008us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:41:53.115202 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:41:53.115383 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:41:53.115476 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:53.115481 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:53.116595 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:53.116599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:53.117381 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:41:53.117393 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:41:53.118001 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:41:53.118005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:41:53.118089 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:53.118572 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:41:53.118587 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:41:53.120047 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:41:53.122112 (+ 2065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:41:53.122115 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:41:53.122118 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:41:53.122362 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:41:53.122875 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:41:53.122878 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:41:53.122880 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:41:53.122976 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:41:53.123538 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:41:53.123541 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:41:53.123644 (+ 103us) client_negotiation.cc:769] Sending connection context
0619 14:41:53.123724 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 14:41:53.123888 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155}
I20260619 14:41:53.124199 23261 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74d0d6c4050f45888d6e6eed817112e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:41:53.124859 23186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5a0a4fea8d447788d89d9093e36f4bb: No bootstrap required, opened a new log
I20260619 14:41:53.125211 23266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:53.114831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48058 (local address 127.0.98.60:39047)
0619 14:41:53.114972 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:53.114982 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:53.114995 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:41:53.115042 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:53.115044 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:53.115082 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:53.115159 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:53.115618 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:53.116392 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:53.117519 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:53.117848 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:53.117983 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:53.120200 (+ 2217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:53.120204 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:53.120207 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:41:53.120229 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:53.121991 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:53.122624 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:53.122627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:53.122629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:53.122770 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:53.123191 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:53.123194 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:53.123196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:53.123410 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:53.124774 (+ 1364us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:53.124861 (+ 87us) server_negotiation.cc:299] Negotiation successful
0619 14:41:53.125040 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 14:41:53.128571 23111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 170742111f2f4017a44c972bdbd2e3bb: Bootstrap starting.
I20260619 14:41:53.128919 23186 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5a0a4fea8d447788d89d9093e36f4bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170742111f2f4017a44c972bdbd2e3bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40495 } } peers { permanent_uuid: "c5a0a4fea8d447788d89d9093e36f4bb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33015 } } peers { permanent_uuid: "74d0d6c4050f45888d6e6eed817112e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39047 } }
I20260619 14:41:53.129153 23186 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5a0a4fea8d447788d89d9093e36f4bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5a0a4fea8d447788d89d9093e36f4bb, State: Initialized, Role: FOLLOWER
I20260619 14:41:53.129531 23186 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5a0a4fea8d447788d89d9093e36f4bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170742111f2f4017a44c972bdbd2e3bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40495 } } peers { permanent_uuid: "c5a0a4fea8d447788d89d9093e36f4bb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33015 } } peers { permanent_uuid: "74d0d6c4050f45888d6e6eed817112e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39047 } }
I20260619 14:41:53.130070 23282 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5a0a4fea8d447788d89d9093e36f4bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170742111f2f4017a44c972bdbd2e3bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40495 } } peers { permanent_uuid: "c5a0a4fea8d447788d89d9093e36f4bb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33015 } } peers { permanent_uuid: "74d0d6c4050f45888d6e6eed817112e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39047 } } }
I20260619 14:41:53.130312 23282 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5a0a4fea8d447788d89d9093e36f4bb [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:41:53.130820 23186 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5a0a4fea8d447788d89d9093e36f4bb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:41:53.131253 23111 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 170742111f2f4017a44c972bdbd2e3bb: Neither blocks nor log segments found. Creating new log.
I20260619 14:41:53.132135 23111 log.cc:824] T 00000000000000000000000000000000 P 170742111f2f4017a44c972bdbd2e3bb: Log is configured to *not* fsync() on all Append() calls
W20260619 14:41:53.129650 23280 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 74d0d6c4050f45888d6e6eed817112e5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:41:53.133028 23280 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:41:53.134744 23111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 170742111f2f4017a44c972bdbd2e3bb: No bootstrap required, opened a new log
I20260619 14:41:53.134536 23192 master_runner.cc:186] Error getting master registration for 127.0.98.62:40495: OK, instance_id { permanent_uuid: "170742111f2f4017a44c972bdbd2e3bb" instance_seqno: 1781880112650529 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 14:41:53.137748 23111 raft_consensus.cc:348] T 00000000000000000000000000000000 P 170742111f2f4017a44c972bdbd2e3bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170742111f2f4017a44c972bdbd2e3bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40495 } } peers { permanent_uuid: "c5a0a4fea8d447788d89d9093e36f4bb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33015 } } peers { permanent_uuid: "74d0d6c4050f45888d6e6eed817112e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39047 } }
I20260619 14:41:53.137920 23192 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 14:41:53.137957 23111 raft_consensus.cc:729] T 00000000000000000000000000000000 P 170742111f2f4017a44c972bdbd2e3bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 170742111f2f4017a44c972bdbd2e3bb, State: Initialized, Role: FOLLOWER
I20260619 14:41:53.134488 23118 master_runner.cc:186] Error getting master registration for 127.0.98.62:40495: OK, instance_id { permanent_uuid: "170742111f2f4017a44c972bdbd2e3bb" instance_seqno: 1781880112650529 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 14:41:53.138299 23118 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 14:41:53.138278 23111 consensus_queue.cc:260] T 00000000000000000000000000000000 P 170742111f2f4017a44c972bdbd2e3bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170742111f2f4017a44c972bdbd2e3bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40495 } } peers { permanent_uuid: "c5a0a4fea8d447788d89d9093e36f4bb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33015 } } peers { permanent_uuid: "74d0d6c4050f45888d6e6eed817112e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39047 } }
I20260619 14:41:53.139601 23111 sys_catalog.cc:564] T 00000000000000000000000000000000 P 170742111f2f4017a44c972bdbd2e3bb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:41:53.139978 23085 raft_consensus.cc:482] T 00000000000000000000000000000000 P 170742111f2f4017a44c972bdbd2e3bb [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 14:41:53.140093 23085 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 170742111f2f4017a44c972bdbd2e3bb [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:41:53.141405 23085 raft_consensus.cc:504] T 00000000000000000000000000000000 P 170742111f2f4017a44c972bdbd2e3bb [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170742111f2f4017a44c972bdbd2e3bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40495 } } peers { permanent_uuid: "c5a0a4fea8d447788d89d9093e36f4bb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33015 } } peers { permanent_uuid: "74d0d6c4050f45888d6e6eed817112e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39047 } }
W20260619 14:41:53.141601 23294 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c5a0a4fea8d447788d89d9093e36f4bb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:41:53.141659 23294 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:41:53.142417 23085 leader_election.cc:288] T 00000000000000000000000000000000 P 170742111f2f4017a44c972bdbd2e3bb [CANDIDATE]: Term 1 election: Requested vote from peers c5a0a4fea8d447788d89d9093e36f4bb (127.0.98.61:33015), 74d0d6c4050f45888d6e6eed817112e5 (127.0.98.60:39047)
I20260619 14:41:53.143344 23160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "170742111f2f4017a44c972bdbd2e3bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c5a0a4fea8d447788d89d9093e36f4bb"
I20260619 14:41:53.143376 23235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "170742111f2f4017a44c972bdbd2e3bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "74d0d6c4050f45888d6e6eed817112e5"
I20260619 14:41:53.143586 23235 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74d0d6c4050f45888d6e6eed817112e5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:41:53.143589 23160 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5a0a4fea8d447788d89d9093e36f4bb [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:41:53.144865 23235 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 74d0d6c4050f45888d6e6eed817112e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 170742111f2f4017a44c972bdbd2e3bb in term 1.
I20260619 14:41:53.144870 23160 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c5a0a4fea8d447788d89d9093e36f4bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 170742111f2f4017a44c972bdbd2e3bb in term 1.
I20260619 14:41:53.144970 23295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 170742111f2f4017a44c972bdbd2e3bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170742111f2f4017a44c972bdbd2e3bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40495 } } peers { permanent_uuid: "c5a0a4fea8d447788d89d9093e36f4bb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33015 } } peers { permanent_uuid: "74d0d6c4050f45888d6e6eed817112e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39047 } } }
I20260619 14:41:53.145092 23295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 170742111f2f4017a44c972bdbd2e3bb [sys.catalog]: This master's current role is: FOLLOWER
W20260619 14:41:53.147456 23064 tablet.cc:2369] T 00000000000000000000000000000000 P 170742111f2f4017a44c972bdbd2e3bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:41:53.148370 23059 leader_election.cc:302] T 00000000000000000000000000000000 P 170742111f2f4017a44c972bdbd2e3bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 170742111f2f4017a44c972bdbd2e3bb, 74d0d6c4050f45888d6e6eed817112e5, c5a0a4fea8d447788d89d9093e36f4bb; no voters:
I20260619 14:41:53.148631 23295 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 170742111f2f4017a44c972bdbd2e3bb [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:41:53.148860 23295 raft_consensus.cc:686] T 00000000000000000000000000000000 P 170742111f2f4017a44c972bdbd2e3bb [term 1 LEADER]: Becoming Leader. State: Replica: 170742111f2f4017a44c972bdbd2e3bb, State: Running, Role: LEADER
I20260619 14:41:53.149199 23295 consensus_queue.cc:237] T 00000000000000000000000000000000 P 170742111f2f4017a44c972bdbd2e3bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170742111f2f4017a44c972bdbd2e3bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40495 } } peers { permanent_uuid: "c5a0a4fea8d447788d89d9093e36f4bb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33015 } } peers { permanent_uuid: "74d0d6c4050f45888d6e6eed817112e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39047 } }
I20260619 14:41:53.150954 23295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 170742111f2f4017a44c972bdbd2e3bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 170742111f2f4017a44c972bdbd2e3bb. Latest consensus state: current_term: 1 leader_uuid: "170742111f2f4017a44c972bdbd2e3bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170742111f2f4017a44c972bdbd2e3bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40495 } } peers { permanent_uuid: "c5a0a4fea8d447788d89d9093e36f4bb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33015 } } peers { permanent_uuid: "74d0d6c4050f45888d6e6eed817112e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39047 } } }
I20260619 14:41:53.151190 23295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 170742111f2f4017a44c972bdbd2e3bb [sys.catalog]: This master's current role is: LEADER
I20260619 14:41:53.153581 23043 master_runner.cc:263] No leader master found from master 170742111f2f4017a44c972bdbd2e3bb
I20260619 14:41:53.153642 23043 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 14:41:53.157018 23317 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:41:53.159627 23114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:53.146861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36252 (local address 127.0.98.62:40495)
0619 14:41:53.146965 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:53.146969 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:53.149474 (+ 2505us) server_negotiation.cc:407] Connection header received
0619 14:41:53.149856 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:53.149859 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:53.149904 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:53.149950 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:53.151980 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:53.152790 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:53.153974 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:53.154328 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:53.154429 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:53.156134 (+ 1705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:53.156138 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:53.156141 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:41:53.156166 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:53.157921 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:53.158374 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:53.158378 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:53.158380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:53.158458 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:53.158731 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:53.158734 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:53.158736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:53.158933 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:53.159028 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:53.159360 (+ 332us) server_negotiation.cc:299] Negotiation successful
0619 14:41:53.159472 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 14:41:53.160210 23317 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:41:53.169118 23266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:53.148766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48068 (local address 127.0.98.60:39047)
0619 14:41:53.148875 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:53.148879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:53.160494 (+ 11615us) server_negotiation.cc:407] Connection header received
0619 14:41:53.160647 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:53.160650 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:53.160699 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:53.160775 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:53.161597 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:53.162408 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:53.163511 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:53.163881 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:53.163954 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:53.165600 (+ 1646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:53.165604 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:53.165607 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:41:53.165636 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:53.167314 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:53.167756 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:53.167760 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:53.167762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:53.167852 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:53.168146 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:53.168150 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:53.168152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:53.168372 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:53.168473 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:53.168820 (+ 347us) server_negotiation.cc:299] Negotiation successful
0619 14:41:53.168937 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 14:41:53.171923 23191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:53.147470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45888 (local address 127.0.98.61:33015)
0619 14:41:53.147590 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:53.147595 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:53.156985 (+ 9390us) server_negotiation.cc:407] Connection header received
0619 14:41:53.157168 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:53.157172 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:53.157228 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:53.157277 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:53.164902 (+ 7625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:53.165662 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:53.166769 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:53.167087 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:53.167187 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:53.168707 (+ 1520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:41:53.168711 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:41:53.168714 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:41:53.168762 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:41:53.170226 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:53.170688 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:53.170691 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:53.170693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:53.170777 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:41:53.171037 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:41:53.171040 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:41:53.171041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:41:53.171248 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:41:53.171356 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:53.171662 (+ 306us) server_negotiation.cc:299] Negotiation successful
0619 14:41:53.171773 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 14:41:53.176155 23235 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 74d0d6c4050f45888d6e6eed817112e5 [term 1 FOLLOWER]: Refusing update from remote peer 170742111f2f4017a44c972bdbd2e3bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:41:53.176242 23160 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c5a0a4fea8d447788d89d9093e36f4bb [term 1 FOLLOWER]: Refusing update from remote peer 170742111f2f4017a44c972bdbd2e3bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:41:53.176924 23314 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 170742111f2f4017a44c972bdbd2e3bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5a0a4fea8d447788d89d9093e36f4bb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:41:53.180684 23282 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5a0a4fea8d447788d89d9093e36f4bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 170742111f2f4017a44c972bdbd2e3bb. Latest consensus state: current_term: 1 leader_uuid: "170742111f2f4017a44c972bdbd2e3bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170742111f2f4017a44c972bdbd2e3bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40495 } } peers { permanent_uuid: "c5a0a4fea8d447788d89d9093e36f4bb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33015 } } peers { permanent_uuid: "74d0d6c4050f45888d6e6eed817112e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39047 } } }
I20260619 14:41:53.180825 23282 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5a0a4fea8d447788d89d9093e36f4bb [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:41:53.180938 23314 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 170742111f2f4017a44c972bdbd2e3bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "74d0d6c4050f45888d6e6eed817112e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:41:53.184630 23269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74d0d6c4050f45888d6e6eed817112e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 170742111f2f4017a44c972bdbd2e3bb. Latest consensus state: current_term: 1 leader_uuid: "170742111f2f4017a44c972bdbd2e3bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170742111f2f4017a44c972bdbd2e3bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40495 } } peers { permanent_uuid: "c5a0a4fea8d447788d89d9093e36f4bb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33015 } } peers { permanent_uuid: "74d0d6c4050f45888d6e6eed817112e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39047 } } }
I20260619 14:41:53.184777 23269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74d0d6c4050f45888d6e6eed817112e5 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:41:53.191243 23314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 170742111f2f4017a44c972bdbd2e3bb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "170742111f2f4017a44c972bdbd2e3bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170742111f2f4017a44c972bdbd2e3bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40495 } } peers { permanent_uuid: "c5a0a4fea8d447788d89d9093e36f4bb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33015 } } peers { permanent_uuid: "74d0d6c4050f45888d6e6eed817112e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39047 } } }
I20260619 14:41:53.191383 23314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 170742111f2f4017a44c972bdbd2e3bb [sys.catalog]: This master's current role is: LEADER
I20260619 14:41:53.192541 23282 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5a0a4fea8d447788d89d9093e36f4bb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "170742111f2f4017a44c972bdbd2e3bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170742111f2f4017a44c972bdbd2e3bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40495 } } peers { permanent_uuid: "c5a0a4fea8d447788d89d9093e36f4bb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33015 } } peers { permanent_uuid: "74d0d6c4050f45888d6e6eed817112e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39047 } } }
I20260619 14:41:53.192636 23282 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5a0a4fea8d447788d89d9093e36f4bb [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:41:53.192924 23320 mvcc.cc:205] Tried to move back new op lower bound from 7298580943562600448 to 7298580943462330368. Current Snapshot: MvccSnapshot[applied={T|T < 7298580943562600448}]
I20260619 14:41:53.193082 23295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 170742111f2f4017a44c972bdbd2e3bb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "170742111f2f4017a44c972bdbd2e3bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170742111f2f4017a44c972bdbd2e3bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40495 } } peers { permanent_uuid: "c5a0a4fea8d447788d89d9093e36f4bb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33015 } } peers { permanent_uuid: "74d0d6c4050f45888d6e6eed817112e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39047 } } }
I20260619 14:41:53.193197 23295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 170742111f2f4017a44c972bdbd2e3bb [sys.catalog]: This master's current role is: LEADER
I20260619 14:41:53.194156 23269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74d0d6c4050f45888d6e6eed817112e5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "170742111f2f4017a44c972bdbd2e3bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170742111f2f4017a44c972bdbd2e3bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40495 } } peers { permanent_uuid: "c5a0a4fea8d447788d89d9093e36f4bb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33015 } } peers { permanent_uuid: "74d0d6c4050f45888d6e6eed817112e5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39047 } } }
I20260619 14:41:53.194265 23269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74d0d6c4050f45888d6e6eed817112e5 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:41:53.193120 23319 mvcc.cc:205] Tried to move back new op lower bound from 7298580943562600448 to 7298580943462330368. Current Snapshot: MvccSnapshot[applied={T|T < 7298580943562600448}]
I20260619 14:41:53.194852 23322 mvcc.cc:205] Tried to move back new op lower bound from 7298580943562600448 to 7298580943462330368. Current Snapshot: MvccSnapshot[applied={T|T < 7298580943562600448}]
I20260619 14:41:53.196436 23317 catalog_manager.cc:1347] Generated new cluster ID: e6b9c46f43644d8cae3bb9f95e4251e2
I20260619 14:41:53.196496 23317 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:41:53.220037 23317 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:41:53.222467 23317 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:41:53.243827 23317 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 170742111f2f4017a44c972bdbd2e3bb: Generated new TSK 0
I20260619 14:41:53.244771 23317 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 14:41:53.272492 23114 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:41:53.269560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36260 (local address 127.0.98.62:40495)
0619 14:41:53.269687 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:53.269692 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:53.271457 (+ 1765us) server_negotiation.cc:407] Connection header received
0619 14:41:53.271650 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:53.271653 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:53.271703 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:53.271750 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:53.272339 (+ 589us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36260: BlockingRecv error: recv got EOF from 127.0.0.1:36260 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260619 14:41:53.273816 23266 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:41:53.269990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48072 (local address 127.0.98.60:39047)
0619 14:41:53.270136 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:53.270140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:53.272969 (+ 2829us) server_negotiation.cc:407] Connection header received
0619 14:41:53.273120 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:53.273122 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:53.273167 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:53.273211 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:53.273698 (+ 487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48072: BlockingRecv error: recv got EOF from 127.0.0.1:48072 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260619 14:41:53.275086 23191 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:41:53.270667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45896 (local address 127.0.98.61:33015)
0619 14:41:53.270776 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:53.270780 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:53.274241 (+ 3461us) server_negotiation.cc:407] Connection header received
0619 14:41:53.274397 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:53.274400 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:53.274464 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:53.274517 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:53.274980 (+ 463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45896: BlockingRecv error: recv got EOF from 127.0.0.1:45896 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260619 14:41:53.279214 23191 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:41:53.277969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45910 (local address 127.0.98.61:33015)
0619 14:41:53.278084 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:53.278087 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:53.278305 (+ 218us) server_negotiation.cc:407] Connection header received
0619 14:41:53.278524 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:53.278527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:53.278599 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:53.278649 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:53.279120 (+ 471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45910: BlockingRecv error: recv got EOF from 127.0.0.1:45910 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260619 14:41:53.283924 23191 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:41:53.282472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45916 (local address 127.0.98.61:33015)
0619 14:41:53.282564 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:53.282568 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:53.283101 (+ 533us) server_negotiation.cc:407] Connection header received
0619 14:41:53.283270 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:53.283273 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:53.283319 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:53.283365 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:53.283835 (+ 470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45916: BlockingRecv error: recv got EOF from 127.0.0.1:45916 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260619 14:41:53.289518 23191 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:41:53.288069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45920 (local address 127.0.98.61:33015)
0619 14:41:53.288162 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:53.288166 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:53.288642 (+ 476us) server_negotiation.cc:407] Connection header received
0619 14:41:53.288842 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:53.288845 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:53.288895 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:53.288943 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:53.289421 (+ 478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45920: BlockingRecv error: recv got EOF from 127.0.0.1:45920 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260619 14:41:53.297389 23191 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:41:53.295707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45928 (local address 127.0.98.61:33015)
0619 14:41:53.295840 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:53.295844 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:53.296423 (+ 579us) server_negotiation.cc:407] Connection header received
0619 14:41:53.296630 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:53.296633 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:53.296685 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:53.296771 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:53.297278 (+ 507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45928: BlockingRecv error: recv got EOF from 127.0.0.1:45928 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260619 14:41:53.309504 23191 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:41:53.307773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45940 (local address 127.0.98.61:33015)
0619 14:41:53.307886 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:53.307890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:53.308412 (+ 522us) server_negotiation.cc:407] Connection header received
0619 14:41:53.308652 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:53.308656 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:53.308714 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:53.308812 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:53.309377 (+ 565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45940: BlockingRecv error: recv got EOF from 127.0.0.1:45940 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260619 14:41:53.330327 23191 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:41:53.328349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45950 (local address 127.0.98.61:33015)
0619 14:41:53.328486 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:53.328491 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:53.329127 (+ 636us) server_negotiation.cc:407] Connection header received
0619 14:41:53.329427 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:53.329432 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:53.329506 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:53.329563 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:53.330176 (+ 613us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45950: BlockingRecv error: recv got EOF from 127.0.0.1:45950 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260619 14:41:53.365835 23191 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:41:53.364634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45966 (local address 127.0.98.61:33015)
0619 14:41:53.364788 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:53.364792 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:53.365001 (+ 209us) server_negotiation.cc:407] Connection header received
0619 14:41:53.365119 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:53.365123 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:53.365182 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:53.365269 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:53.365722 (+ 453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45966: BlockingRecv error: recv got EOF from 127.0.0.1:45966 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260619 14:41:53.432973 23191 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:41:53.431815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45972 (local address 127.0.98.61:33015)
0619 14:41:53.431943 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:53.431948 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:53.432095 (+ 147us) server_negotiation.cc:407] Connection header received
0619 14:41:53.432225 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:53.432228 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:53.432278 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:53.432355 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:53.432840 (+ 485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45972: BlockingRecv error: recv got EOF from 127.0.0.1:45972 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260619 14:41:53.564071 23191 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:41:53.562912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45988 (local address 127.0.98.61:33015)
0619 14:41:53.563097 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:53.563102 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:53.563226 (+ 124us) server_negotiation.cc:407] Connection header received
0619 14:41:53.563375 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:53.563379 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:53.563428 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:53.563500 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:53.563951 (+ 451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45988: BlockingRecv error: recv got EOF from 127.0.0.1:45988 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260619 14:41:53.823117 23191 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:41:53.822115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45996 (local address 127.0.98.61:33015)
0619 14:41:53.822225 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:53.822228 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:53.822363 (+ 135us) server_negotiation.cc:407] Connection header received
0619 14:41:53.822461 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:53.822464 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:53.822516 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:53.822597 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:41:53.822999 (+ 402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45996: BlockingRecv error: recv got EOF from 127.0.0.1:45996 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 14:41:54.134294 23280 catalog_manager.cc:1259] Loaded cluster ID: e6b9c46f43644d8cae3bb9f95e4251e2
I20260619 14:41:54.134379 23280 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 74d0d6c4050f45888d6e6eed817112e5: loading cluster ID for follower catalog manager: success
I20260619 14:41:54.136152 23280 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 74d0d6c4050f45888d6e6eed817112e5: acquiring CA information for follower catalog manager: success
I20260619 14:41:54.137403 23280 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 74d0d6c4050f45888d6e6eed817112e5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 14:41:54.142915 23294 catalog_manager.cc:1259] Loaded cluster ID: e6b9c46f43644d8cae3bb9f95e4251e2
I20260619 14:41:54.142982 23294 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c5a0a4fea8d447788d89d9093e36f4bb: loading cluster ID for follower catalog manager: success
I20260619 14:41:54.144683 23294 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c5a0a4fea8d447788d89d9093e36f4bb: acquiring CA information for follower catalog manager: success
I20260619 14:41:54.146052 23294 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c5a0a4fea8d447788d89d9093e36f4bb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 14:41:54.340859 23393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:54.337190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46008 (local address 127.0.98.61:33015)
0619 14:41:54.337431 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:54.337435 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:54.337495 (+ 60us) server_negotiation.cc:407] Connection header received
0619 14:41:54.337618 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:54.337620 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:54.337746 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:54.337850 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:41:54.338053 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:54.338581 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:54.339432 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:54.339652 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:54.339725 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:54.339998 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:41:54.340432 (+ 434us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:41:54.340525 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:54.340619 (+ 94us) server_negotiation.cc:299] Negotiation successful
0619 14:41:54.340669 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":68,"threads_started":1}
I20260619 14:41:54.347637 23399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:54.343877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48080 (local address 127.0.98.60:39047)
0619 14:41:54.344112 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:54.344117 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:54.344214 (+ 97us) server_negotiation.cc:407] Connection header received
0619 14:41:54.344361 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:54.344364 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:54.344479 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:54.344576 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:41:54.344894 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:54.345425 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:54.346250 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:54.346468 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:54.346551 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:54.346833 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:41:54.347265 (+ 432us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:41:54.347373 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:54.347428 (+ 55us) server_negotiation.cc:299] Negotiation successful
0619 14:41:54.347481 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":72,"threads_started":1}
I20260619 14:41:54.354333 23399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:54.350437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48082 (local address 127.0.98.60:39047)
0619 14:41:54.350547 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:54.350551 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:54.350841 (+ 290us) server_negotiation.cc:407] Connection header received
0619 14:41:54.350998 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:54.351000 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:54.351057 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:54.351126 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:41:54.351377 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:54.352139 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:54.353257 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:54.353565 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:54.353653 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:54.353894 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:41:54.353969 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:41:54.354041 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:54.354152 (+ 111us) server_negotiation.cc:299] Negotiation successful
0619 14:41:54.354203 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 14:41:54.354588 23393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:54.350935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46024 (local address 127.0.98.61:33015)
0619 14:41:54.351044 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:54.351047 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:54.351057 (+ 10us) server_negotiation.cc:407] Connection header received
0619 14:41:54.351114 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:54.351117 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:54.351166 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:54.351237 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:41:54.351670 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:54.352382 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:54.353449 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:54.353763 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:54.353854 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:54.354151 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:41:54.354225 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:41:54.354288 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:54.354424 (+ 136us) server_negotiation.cc:299] Negotiation successful
0619 14:41:54.354480 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 14:41:54.355348 23407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:41:54.350867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36264 (local address 127.0.98.62:40495)
0619 14:41:54.351077 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 14:41:54.351081 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:41:54.351093 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:41:54.351176 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:41:54.351179 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:41:54.351328 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:41:54.351445 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:41:54.351647 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:54.352156 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:54.353116 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:41:54.353424 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:41:54.353527 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:41:54.353824 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:41:54.354146 (+ 322us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:41:54.354810 (+ 664us) server_negotiation.cc:1035] Waiting for connection context
0619 14:41:54.354853 (+ 43us) server_negotiation.cc:299] Negotiation successful
0619 14:41:54.354913 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":58,"threads_started":1}
I20260619 14:41:54.358100 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 23043
I20260619 14:41:54.363709 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 23118
I20260619 14:41:54.368551 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7PHJY_/build/debug/bin/kudu with pid 23192
2026-06-19T14:41:54Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2967 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5964 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (276712 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST